| *** tosky has quit IRC | 00:01 | |
| *** dsneddon has quit IRC | 00:05 | |
| *** dsneddon has joined #oooq | 00:06 | |
| *** rlandy|ruck has quit IRC | 00:24 | |
| *** dsneddon has quit IRC | 00:31 | |
| *** dsneddon has joined #oooq | 00:54 | |
| *** dsneddon has quit IRC | 00:59 | |
| *** apetrich has quit IRC | 01:59 | |
| *** weshay_pto is now known as weshay | 02:49 | |
| *** dsneddon has joined #oooq | 03:03 | |
| *** dsneddon has quit IRC | 03:12 | |
| *** dsneddon has joined #oooq | 03:41 | |
| *** dsneddon has quit IRC | 03:47 | |
| *** dsneddon has joined #oooq | 04:12 | |
| *** ratailor has joined #oooq | 04:16 | |
| *** ykarel|away has joined #oooq | 04:16 | |
| *** dsneddon has quit IRC | 04:17 | |
| *** ykarel_ has joined #oooq | 04:22 | |
| *** dsneddon has joined #oooq | 04:22 | |
| *** ykarel|away has quit IRC | 04:24 | |
| *** dsneddon has quit IRC | 04:27 | |
| *** dsneddon has joined #oooq | 05:02 | |
| *** dsneddon has quit IRC | 05:15 | |
| *** dsneddon has joined #oooq | 05:15 | |
| *** dsneddon has quit IRC | 05:20 | |
| *** jbadiapa has quit IRC | 05:25 | |
| *** ykarel_ is now known as ykarel | 05:31 | |
| *** ratailor_ has joined #oooq | 05:34 | |
| *** ratailor has quit IRC | 05:36 | |
| *** quiquell|off is now known as quiquell|rover | 05:49 | |
| *** dsneddon has joined #oooq | 05:49 | |
| *** dsneddon has quit IRC | 05:54 | |
| *** marios has joined #oooq | 05:54 | |
| quiquell|rover | https://bugs.launchpad.net/bugs/1824243 | 05:56 |
|---|---|---|
| openstack | Launchpad bug 1824243 in tripleo "cirros image missing from libvirt reproducer" [High,In progress] - Assigned to wes hayutin (weshayutin) | 05:56 |
| quiquell|rover | sshnaidm: can we close it? | 05:56 |
| quiquell|rover | It should work with your latest review | 05:56 |
| *** kopecmartin|off is now known as kopecmartin | 06:07 | |
| *** dsneddon has joined #oooq | 06:20 | |
| *** saneax has joined #oooq | 06:21 | |
| *** dsneddon has quit IRC | 06:25 | |
| *** saneax has quit IRC | 06:25 | |
| *** saneax has joined #oooq | 06:26 | |
| *** sanjayu_ has joined #oooq | 06:28 | |
| *** dsneddon has joined #oooq | 06:40 | |
| *** dsneddon has quit IRC | 06:45 | |
| *** apetrich has joined #oooq | 06:55 | |
| *** marios has quit IRC | 06:55 | |
| *** quiquell|rover is now known as quique|rover|brb | 06:55 | |
| *** marios has joined #oooq | 06:59 | |
| marios | quique|rover|brb: o/ | 07:08 |
| marios | ping when you're back. i think we can merge that now. we discussed in yesterday scrum too and rlandy asked us to tell you before we go. do you want to review it too? | 07:09 |
| marios | https://review.openstack.org/#/c/651362/ that one quique|rover|brb | 07:09 |
| *** quique|rover|brb is now known as quiquell|rover | 07:09 | |
| quiquell|rover | Checking | 07:09 |
| ykarel | quiquell|rover, RDO phase1 pike job running for long, seems stuck at collect-logs | 07:10 |
| ykarel | https://ci.centos.org/job/tripleo-quickstart-promote-pike-rdo_trunk-minimal/240/console | 07:11 |
| quiquell|rover | ykarel: will check | 07:13 |
| ykarel | quiquell|rover, ack, i have seen this behavior long time ago as well | 07:15 |
| quiquell|rover | marios: don't we wsant tripleo-build-containers-centos-7 to run on master and stein ? | 07:15 |
| quiquell|rover | marios: I mean branches: master | 07:15 |
| quiquell|rover | marios: if you make a change a tht stable/stein (when it's there the branch) | 07:16 |
| quiquell|rover | marios: Don't know if -stein job is going to checkout the correct change | 07:16 |
| quiquell|rover | marios: or just the branch | 07:16 |
| marios | quiquell|rover: ack add comment on the review. i'll check in a bit. if i recall folco is using branch overrides there for the stein ones | 07:16 |
| marios | quiquell|rover: yeah we want the job to run master/stein | 07:17 |
| marios | quiquell|rover: i think we have it rocky too | 07:17 |
| *** dsneddon has joined #oooq | 07:17 | |
| quiquell|rover | marios: but branchful is different is for running not branched projects with specific branhes of branched ones | 07:17 |
| quiquell|rover | marios: commented | 07:18 |
| marios | quiquell|rover: ack revoted for now thanks | 07:19 |
| *** ratailor__ has joined #oooq | 07:20 | |
| quiquell|rover | sshnaidm: https://bugs.launchpad.net/tripleo/+bug/1824954 | 07:20 |
| openstack | Launchpad bug 1824954 in tripleo "zuul reproducer Error validating value 'bmc-template': No images matching {'name': u'bmc-template'}." [High,Triaged] | 07:20 |
| *** ratailor_ has quit IRC | 07:22 | |
| *** tosky has joined #oooq | 07:23 | |
| * marios brb | 07:24 | |
| *** marios has quit IRC | 07:24 | |
| quiquell|rover | ykarel: do you have an rdo cloud tenant ? | 07:24 |
| *** marios has joined #oooq | 07:25 | |
| ykarel | quiquell|rover, you mean account on rdo cloud? | 07:25 |
| quiquell|rover | ykarel: yep | 07:26 |
| ykarel | quiquell|rover, yes i have | 07:26 |
| quiquell|rover | want to check one thing | 07:26 |
| quiquell|rover | can you do an "image list" and see if bmc-template appears there ? | 07:26 |
| ykarel | running | 07:26 |
| ykarel | quiquell|rover, no it's not there | 07:27 |
| *** holser_ has joined #oooq | 07:27 | |
| quiquell|rover | ack | 07:27 |
| *** skramaja has joined #oooq | 07:28 | |
| *** jbadiapa has joined #oooq | 07:31 | |
| quiquell|rover | marios: do you know why we don't use fs039 as criteria for stein ? | 07:31 |
| quiquell|rover | marios: I mean promotion criteria | 07:31 |
| *** ykarel is now known as ykarel|lunch | 07:33 | |
| *** amoralej|off is now known as amoralej | 07:33 | |
| marios | quiquell|rover: no | 07:34 |
| *** dsneddon has quit IRC | 07:41 | |
| *** ccamacho has joined #oooq | 07:48 | |
| *** dsneddon has joined #oooq | 07:53 | |
| chandankumar | quiquell|rover: marios arxcruz https://review.rdoproject.org/r/#/c/20203/ full tempest using os_tempest, feel free to take a look when free | 07:57 |
| arxcruz | chandankumar: so, fs020 has 1500+ tests, and this one have 1100+ tests | 07:58 |
| arxcruz | why the difference? | 07:58 |
| quiquell|rover | ykarel|lunch: phase1 queens is timing out | 08:00 |
| chandankumar | arxcruz: fs020 has telemetry, ironic, networking_l2gw tests | 08:07 |
| *** dsneddon has quit IRC | 08:07 | |
| chandankumar | which are not in standalone one | 08:07 |
| *** dsneddon has joined #oooq | 08:08 | |
| *** gkadam has joined #oooq | 08:09 | |
| *** gkadam has quit IRC | 08:10 | |
| *** bogdando has joined #oooq | 08:11 | |
| *** dsneddon has quit IRC | 08:13 | |
| *** ykarel|lunch is now known as ykarel | 08:30 | |
| *** ratailor__ has quit IRC | 08:30 | |
| ykarel | quiquell|rover, let's rerun queens phase1 | 08:32 |
| ykarel | and see what's going on | 08:32 |
| sshnaidm | ykarel, quiquell|rover I see bmc-template is there | 08:32 |
| sshnaidm | did you bring it back? | 08:32 |
| ykarel | it's not there atleast in my tenant | 08:33 |
| quiquell|rover | sshnaidm: has mark it as shared at nodepool tenant | 08:33 |
| quiquell|rover | sshnaidm: ykarel for example cannot see it | 08:33 |
| ykarel | quiquell|rover, restarted queens phase1:- https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-queens-current-tripleo/243/ | 08:34 |
| ykarel | let's see how it goes | 08:34 |
| quiquell|rover | ack | 08:34 |
| quiquell|rover | sshnaidm: I am trying to reproducer tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 | 08:34 |
| ykarel | quiquell|rover, aborting pike phase1 | 08:34 |
| quiquell|rover | sshnaidm: do we have enough power at our RDO tenants ? | 08:34 |
| sshnaidm | I have bmc-template with different ID then in openstack-nodepool tenant, maybe I uploaded it by my own | 08:35 |
| sshnaidm | quiquell|rover, yes, I reproduced it yesterday | 08:35 |
| sshnaidm | quiquell|rover, fails in overcloud.. | 08:36 |
| quiquell|rover | sshnaidm: Yep I am trying to see why freeipa cannot resolve RDO mirros | 08:36 |
| quiquell|rover | sshnaidm: do you have still the living system ? | 08:36 |
| sshnaidm | quiquell|rover, yeah, give me your key | 08:36 |
| ykarel | quiquell|rover, restarted pike phase1 also:- https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-pike-current-tripleo/275/ | 08:36 |
| quiquell|rover | sshnaidm: https://github.com/qinqon.keys | 08:38 |
| quiquell|rover | ykarel: ack | 08:38 |
| sshnaidm | quiquell|rover, ssh zuul@38.145.34.6 | 08:39 |
| quiquell|rover | Ok I am in | 08:40 |
| quiquell|rover | Let me know if you need your tenant | 08:40 |
| quiquell|rover | I am running the job in parallel at mine | 08:40 |
| *** dsneddon has joined #oooq | 08:45 | |
| quiquell|rover | sshnaidm: do you know how to access the freeipa node ? | 08:46 |
| *** aakarsh has quit IRC | 08:47 | |
| quiquell|rover | Ahh I found the extra rsa key | 08:48 |
| sshnaidm | quiquell|rover, the key is in ~/extranode-id_rsa , the address is in /etc/resolv.conf | 08:48 |
| sshnaidm | quiquell|rover, and user is centos | 08:48 |
| zbr | who can tell me a little bit about how the files from tripleo-ci/toci-quickstart/config/testenv are loaded? | 08:50 |
| quiquell|rover | sshnaidm: humm this error is different | 08:50 |
| *** dsneddon has quit IRC | 08:50 | |
| quiquell|rover | sshnaidm: freeipa is ok there :-( | 08:50 |
| zbr | i was trying to help reviewing https://review.openstack.org/#/c/639008/ and I want to suggest defining the cirros-image in a single/shared place. | 08:51 |
| *** dsneddon has joined #oooq | 08:58 | |
| quiquell|rover | ykarel: can we run this without the Depends-On ? https://review.rdoproject.org/r/#/c/20171/ | 09:01 |
| *** dsneddon has quit IRC | 09:02 | |
| *** dsneddon has joined #oooq | 09:04 | |
| *** dsneddon has quit IRC | 09:09 | |
| ykarel | quiquell|rover, yes why not! | 09:13 |
| quiquell|rover | ykarel: ack doing at other review | 09:14 |
| quiquell|rover | ykarel: just to make sure that it happends with current-tripleo too | 09:15 |
| *** dsneddon has joined #oooq | 09:19 | |
| quiquell|rover | ykarel: I am going to workflow this https://review.openstack.org/#/c/652027/ | 09:23 |
| quiquell|rover | ykarel: I see build containers jobs working fine | 09:23 |
| ykarel | quiquell|rover, yes it's working fine | 09:24 |
| ykarel | i didn't changed end interface | 09:24 |
| ykarel | so should be good | 09:24 |
| quiquell|rover | ok +w | 09:24 |
| ykarel | okk THanks | 09:24 |
| ykarel | Thansk panda | 09:25 |
| panda | ykarel: thanks you, now I can just replicate the playbook skipping tripleo-repos for rhel | 09:25 |
| panda | and move on | 09:25 |
| panda | quiquell|rover: I think the recheck is working now | 09:26 |
| ykarel | panda, yes, on rdo playbook i am using roles like https://review.rdoproject.org/r/#/c/20144/19/playbooks/run-rdoinfo.yaml@270 | 09:26 |
| quiquell|rover | panda: was the pipe the issue ? | 09:26 |
| panda | quiquell|rover: can you recheck the recheck ? | 09:27 |
| quiquell|rover | recheck(recheck) | 09:27 |
| quiquell|rover | panda: pass me the url to internal fs | 09:29 |
| panda | quiquell|rover: the "pipe" is a block style indicator and its technical name is "literal block indicator". THe problem was not the style, it was missing the chomping specification (-) the not automatically add a newline at the end of the last line, that was probbably confusing the regexp. so instead of | I used >-. Probably |- would have worked, but if the regex grows, we'll have to use > anyway to split it | 09:30 |
| panda | into multiple lines. | 09:30 |
| panda | quiquell|rover: I'll pass it internally | 09:30 |
| quiquell|rover | panda: recheck working at one of my reviews | 09:32 |
| quiquell|rover | panda: we are all good now | 09:32 |
| quiquell|rover | thanks | 09:32 |
| panda | \o/ | 09:33 |
| *** dtantsur|afk is now known as dtantsur | 09:35 | |
| *** saneax has quit IRC | 09:39 | |
| *** holser_ is now known as holser|luuunch | 09:53 | |
| quiquell|rover | ykarel, sshnaidm: freeipa is working fine with reproducer :-((( | 09:56 |
| ykarel | :(( | 09:57 |
| quiquell|rover | Resolve correctly the stuff | 09:57 |
| quiquell|rover | zbr: -> https://review.openstack.org/#/c/652066/ | 10:10 |
| quiquell|rover | zbr: what's the issue with removeing reinstall ? | 10:10 |
| quiquell|rover | zbr: now is breaking reproduce fedora28 jobs | 10:10 |
| zbr | quiquell|rover: changed my vote to +1 because is only f28. | 10:11 |
| zbr | is a long and messy story... | 10:12 |
| quiquell|rover | zbr: ack thanks, yep very messy, looks like now jobs are working some maybe they have "uncooked" the zuul images | 10:14 |
| quiquell|rover | panda, sshnaidm: can you merge this ? https://review.openstack.org/#/c/652066 | 10:15 |
| quiquell|rover | It will allow to run fedora28 jobs at reproducer | 10:15 |
| zbr | quiquell|rover: yeah, that is why is messy. | 10:16 |
| sshnaidm | quiquell|rover, np, just let's wait for CI | 10:16 |
| quiquell|rover | sshnaidm: | 10:16 |
| sshnaidm | quiquell|rover, I moved yesterday repro centos7 jobs to vexxhost | 10:17 |
| sshnaidm | quiquell|rover, they have better chance to pass there | 10:17 |
| quiquell|rover | they work ? | 10:17 |
| sshnaidm | quiquell|rover, mostly | 10:17 |
| quiquell|rover | sshnaidm: can we close https://bugs.launchpad.net/tripleo/+bug/1824243 ? | 10:18 |
| openstack | Launchpad bug 1824243 in tripleo "cirros image missing from libvirt reproducer" [High,Fix released] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 10:18 |
| sshnaidm | quiquell|rover, closed | 10:18 |
| zbr | sshnaidm: build results are always schrodinger booleans ;) | 10:19 |
| quiquell|rover | zbr: do you have access to rdo cloud ? | 10:19 |
| quiquell|rover | zbr: can you tell me if you see a pair of reproducer cloudinit images at your tenant ? | 10:19 |
| quiquell|rover | ykarel: ^ maybe you can help me there too | 10:19 |
| quiquell|rover | chandankumar, arxcruz: this is happening again https://bugs.launchpad.net/tripleo/+bug/1824315 | 10:24 |
| openstack | Launchpad bug 1824315 in tripleo "periodic fedora28 standalone job failing at test_volume_boot_pattern" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 10:24 |
| arxcruz | quiquell|rover: yup, by any chance you have a env that i can digg into it ? | 10:24 |
| quiquell|rover | arxcruz: nope, I am testing the freeipa issue | 10:25 |
| chandankumar | arxcruz: https://review.rdoproject.org/r/#/c/20203/ anything more we need for getting this merged | 10:25 |
| chandankumar | ? | 10:25 |
| arxcruz | chandankumar: sorry, i thought i had +1 this one this morning | 10:26 |
| chandankumar | arxcruz: np, thanks! | 10:26 |
| chandankumar | panda: quiquell|rover sshnaidm https://review.rdoproject.org/r/#/c/20203/ and https://review.rdoproject.org/r/#/c/20227/ when free, please have a look, it is good to go | 10:29 |
| quiquell|rover | chandankumar: +w https://review.rdoproject.org/r/#/c/20203/ | 10:33 |
| chandankumar | quiquell|rover: thanks! | 10:34 |
| quiquell|rover | chandankumar: question, why do you want to add it as periodic instead of check ? | 10:34 |
| chandankumar | quiquell|rover: https://review.rdoproject.org/r/#/c/20227/ was written to run full tempest and it takes more than 2 hours | 10:35 |
| chandankumar | so I wanted to add it in periodic | 10:35 |
| quiquell|rover | ok +w too | 10:36 |
| chandankumar | panda: to thanks! | 10:36 |
| sshnaidm | quiquell|rover, do you need still 029 env? | 10:45 |
| sshnaidm | s/029/039/ | 10:45 |
| quiquell|rover | sshnaidm: nope | 10:45 |
| quiquell|rover | sshnaidm: failure there is not the freeipa issue :-/ | 10:45 |
| amoralej | quiquell|rover, we are adding a new package ovn to deps repo in master that will obsolete previous openstack-ovn packages | 10:46 |
| sshnaidm | quiquell|rover, I'm not sure ipa has issue, or maybe it's something transient | 10:46 |
| amoralej | we have gated it with oooq and non-oooq jobs, so i think it should not break any job | 10:46 |
| chandankumar | arxcruz: I have ublocked https://tree.taiga.io/project/tripleo-ci-board/us/867?milestone=226267 os_tempest port upstream standlaone jobs userstroy | 10:46 |
| amoralej | but let us know if something explodes | 10:46 |
| quiquell|rover | amoralej: noted, will keep an eye on it | 10:46 |
| quiquell|rover | sshnaidm: puff don't know... ipa config looks ok is able to resolve names at my reproducer | 10:47 |
| chandankumar | sshnaidm: Hello | 10:47 |
| quiquell|rover | sshnaidm: maybe it has problems when is running at "nodepool" tenant somehow | 10:47 |
| sshnaidm | chandankumar, hi | 10:47 |
| chandankumar | sshnaidm: In last os_tempest meeting, for every new fs we need to use os_tempest for running tempest tests, instrad of validate tempest | 10:47 |
| chandankumar | since fs039 is relatively new, we are more to os_tempest there | 10:47 |
| chandankumar | *instead | 10:48 |
| chandankumar | *moving | 10:48 |
| chandankumar | sshnaidm: if it is ok? | 10:48 |
| sshnaidm | chandankumar, great, but it has problems still with tempest config afaik | 10:48 |
| sshnaidm | chandankumar, yeah, no problems with that generally | 10:48 |
| chandankumar | sshnaidm: we have got that shorted out | 10:48 |
| chandankumar | sshnaidm: https://review.openstack.org/#/c/639324/20/config/general_config/featureset039.yml@151 | 10:49 |
| chandankumar | just waiting for the fix of freeipa timedout issue | 10:49 |
| quiquell|rover | sshnaidm: can be that 10.0.0.250 is pointing to elsewhere at "nodepool" tenant ? | 10:50 |
| sshnaidm | quiquell|rover, nope, it's external network in ovb, used in every ovb stack | 10:50 |
| quiquell|rover | ok | 10:51 |
| sshnaidm | quiquell|rover, maybe it's just perf issue, timeouts or kind of | 10:51 |
| sshnaidm | quiquell|rover, I'd blame networking firstly | 10:51 |
| quiquell|rover | sshnaidm: but the " broken trust chain resolving 'mirror.regionone.rdo-cloud-tripleo.rdoproject.org/AAAA/IN': 38.145.33.91#53" | 10:52 |
| quiquell|rover | sshnaidm: does not look like networking | 10:52 |
| sshnaidm | quiquell|rover, as I saw in one of logs, ipa tries to get answers from our rdo cloud dns and fails, maybe there is a problem | 10:52 |
| sshnaidm | quiquell|rover, yeah, that's what I'm talking about | 10:52 |
| kopecmartin | chandankumar, arxcruz please https://review.openstack.org/#/c/651866/, and three dependent patches as well | 10:52 |
| quiquell|rover | sshnaidm: they say DNS servers are ok | 10:52 |
| quiquell|rover | amoralej: maybe you know if we have some issues with DNS servers at RDO | 10:52 |
| sshnaidm | quiquell|rover, why does it try AAAA instead of A | 10:53 |
| sshnaidm | it's ipv6 | 10:53 |
| sshnaidm | quiquell|rover, 38.145.33.91 is our caching dns server on tripleo-infra tenant | 10:53 |
| quiquell|rover | hummm | 10:53 |
| quiquell|rover | caching dns server ?? | 10:53 |
| sshnaidm | quiquell|rover, yeah, there is dnsmasq running with long cache | 10:54 |
| quiquell|rover | sshnaidm: maybe issue is there | 10:54 |
| sshnaidm | quiquell|rover, we had problems with blocking us from 8.8.8.8 etc | 10:54 |
| quiquell|rover | sshnaidm: reproducer is using the caching server too ? | 10:54 |
| sshnaidm | quiquell|rover, I've checked it actually, resolves w//o problems | 10:54 |
| sshnaidm | quiquell|rover, I think so | 10:54 |
| sshnaidm | quiquell|rover, like every ovb job | 10:55 |
| sshnaidm | quiquell|rover, like any job in rdo cloud | 10:55 |
| quiquell|rover | Then I don't get it | 10:55 |
| sshnaidm | quiquell|rover, does it happen always? | 10:55 |
| quiquell|rover | So AAAA is for IPv6 but we are doing IPv4 | 10:56 |
| quiquell|rover | sshnaidm: it's starting to be more frequencly | 10:56 |
| sshnaidm | quiquell|rover, ipv6 may be fallback if no answer from ipv4 | 10:56 |
| sshnaidm | quiquell|rover, but I don't know how ipa handles this, need help of hrybacki or jaosorior | 10:56 |
| quiquell|rover | sshnaidm: https://softwarefactory-project.io/zuul/t/rdoproject.org/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-master | 10:56 |
| sshnaidm | Unable to establish connection to https://phx2.cloud.rdoproject.org:13000/v3/auth/tokens: HTTPSConnectionPool(host='phx2.cloud.rdoproject.org', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f2f0435f510>: Failed to establish a new connection: [Errno -2] Name or service not known',)) | 10:57 |
| sshnaidm | quiquell|rover, I'd blame networking :) | 10:58 |
| sshnaidm | quiquell|rover, seems like time to call superheroes, like kforde | 10:58 |
| quiquell|rover | sshnaidm: maybe we can remove fs039 from master criteria until this is fixed ? | 11:00 |
| quiquell|rover | sshnaidm: the toher jobs are running ok | 11:00 |
| sshnaidm | quiquell|rover, is it criteria? | 11:00 |
| quiquell|rover | sshnaidm: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/master.ini#L24 | 11:00 |
| quiquell|rover | sshnaidm: that's wrong ? | 11:01 |
| sshnaidm | quiquell|rover, yeah, worth to get it out, but also need to check what the hell is with networking | 11:01 |
| jaosorior | sshnaidm: phx2.cloud.rdoproject.org is not managed by IPA; so it would just forward the request to the dns forwarders that are configured. | 11:02 |
| jaosorior | it does handle IPv6 though, so a AAAA should be fine | 11:02 |
| jaosorior | that is of course assuming that there are forwarders configured at all in IPA | 11:02 |
| *** amoralej is now known as amoralej|lunch | 11:04 | |
| sshnaidm | jaosorior, acc. to logs it sends to 38.145.33.91, it's our dns server in rdo cloud: broken trust chain resolving 'mirror.regionone.rdo-cloud-tripleo.rdoproject.org/AAAA/IN': 38.145.33.91#53" | 11:05 |
| *** panda is now known as panda|lunch | 11:05 | |
| sshnaidm | jaosorior, so it has it as forwarder | 11:05 |
| sshnaidm | jaosorior, but why does it ask for AAAA | 11:05 |
| sshnaidm | jaosorior, can be 2 forwarders configured for ipa in case of one fails? | 11:06 |
| quiquell|rover | sshnaidm: remove fs039 from criteria https://review.rdoproject.org/r/20236 | 11:07 |
| quiquell|rover | panda|lunch: ^ | 11:07 |
| jaosorior | sshnaidm: as part of the installation, we enable --auto-forwarders; this will use whatever DNS servers that were configured on the host as dns forwarders. | 11:15 |
| jaosorior | sshnaidm: so... it all depends on what dns servers were configured in the host to begin with | 11:15 |
| jaosorior | regarding the internals (why it's trying to use AAAA there); I don't know | 11:16 |
| jaosorior | we would need to ask some folks with deeper knowledge on FreeIPA. You could ask rcrit maybe. | 11:17 |
| *** sshnaidm has quit IRC | 11:17 | |
| zbr | who wants to be qe for https://review.openstack.org/#/c/652971/ ? | 11:20 |
| *** quiquell|rover is now known as quique|rover|eat | 11:25 | |
| *** sshnaidm has joined #oooq | 11:26 | |
| sshnaidm | jaosorior, connection was dropped, can you repeat please? | 11:27 |
| *** holser|luuunch is now known as holser_ | 11:27 | |
| jaosorior | sshnaidm: what was the last part you read? | 11:28 |
| jaosorior | 14:15 <jaosorior> sshnaidm: as part of the installation, we enable --auto-forwarders; this will use whatever DNS servers that were configured on the host as dns forwarders. | 11:29 |
| jaosorior | 14:15 <jaosorior> sshnaidm: so... it all depends on what dns servers were configured in the host to begin with | 11:29 |
| jaosorior | 14:16 <jaosorior> regarding the internals (why it's trying to use AAAA there); I don't know | 11:29 |
| jaosorior | 14:16 <jaosorior> we would need to ask some folks with deeper knowledge on FreeIPA. You could ask rcrit maybe | 11:29 |
| sshnaidm | jaosorior, ok, so we have one forwarder currently.. if I add another to /etc/resolv.conf, will it go there if first is timeouted? | 11:30 |
| sshnaidm | jaosorior, do you know if it's possible to make longer timeouts for resolving in ipa? | 11:30 |
| jaosorior | I don't know. rcrit might though. | 11:30 |
| sshnaidm | jaosorior, in which channel he is? | 11:32 |
| jaosorior | oh, I had also sent you a message about that (PM) | 11:33 |
| *** ccamacho has quit IRC | 11:34 | |
| weshay | https://review.openstack.org/#/c/652124/ | 11:35 |
| weshay | morning | 11:44 |
| *** dtantsur is now known as dtantsur|brb | 11:45 | |
| *** quique|rover|eat is now known as quiquell|rover | 12:07 | |
| quiquell|rover | weshay: this will fix it I think https://review.rdoproject.org/r/#/c/20212/ | 12:08 |
| quiquell|rover | weshay: is already merged | 12:08 |
| weshay | k.. thanks | 12:10 |
| zbr | weshay: please have a look at https://review.openstack.org/#/c/652971/3 and tell me if that it was what you would have expected from testing install-deps.sh | 12:14 |
| quiquell|rover | weshay: have remove fs039 from master criteria | 12:14 |
| quiquell|rover | arxcruz: looks like it's related with this http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-standalone-master/36994ee/logs/undercloud/var/log/extra/failed_containers.log.txt.gz | 12:15 |
| quiquell|rover | arxcruz: talking about https://bugs.launchpad.net/tripleo/+bug/1824315 | 12:16 |
| openstack | Launchpad bug 1824315 in tripleo "periodic fedora28 standalone job failing at test_volume_boot_pattern" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 12:16 |
| weshay | quiquell|rover I saw | 12:16 |
| weshay | quiquell|rover sshnaidm is 039 moved to non-voting in 3rd party? | 12:16 |
| quiquell|rover | weshay: kforde is looking into it | 12:16 |
| arxcruz | quiquell|rover: ack | 12:16 |
| sshnaidm | weshay, "non-voting"? | 12:17 |
| sshnaidm | weshay, it's excluded from criteria temporarly | 12:17 |
| weshay | sshnaidm in periodic, but it also runs check no? | 12:17 |
| weshay | 3rd party check | 12:18 |
| sshnaidm | weshay, nothing from 3party votes | 12:19 |
| sshnaidm | weshay, there are no voting and non-voting in 3party | 12:19 |
| weshay | sshnaidm what is that little +1 next rdo 3rd party when all the jobs pass? | 12:19 |
| *** rlandy has joined #oooq | 12:19 | |
| *** rlandy is now known as rlandy|ruck | 12:20 | |
| weshay | sshnaidm I would refer to that as a vote | 12:20 |
| weshay | sshnaidm and if we removed 39 from promotion critieria | 12:20 |
| sshnaidm | weshay, it's the same as was before, if any of jobs fails it's -1 | 12:20 |
| weshay | 39 now is either removed from check, or marked non-voting | 12:20 |
| weshay | sshnaidm am I making sense? | 12:20 |
| rlandy|ruck | quiquell|rover: hey | 12:21 |
| rlandy|ruck | quiquell|rover: how goes it today? | 12:22 |
| sshnaidm | weshay, I'm not sure you can make it "non-voting" | 12:22 |
| weshay | sshnaidm I think it's the same as upstream.. in the job def in zuul, voting: false | 12:22 |
| *** panda|lunch is now known as panda | 12:22 | |
| weshay | and it will still run, if the other jobs pass you get a +1 | 12:22 |
| quiquell|rover | rlandy|ruck: so gates are good now | 12:22 |
| sshnaidm | weshay, and why to do it? it's probably problem with networking | 12:22 |
| quiquell|rover | rlandy|ruck: and I ahve remove fs039 from criteria | 12:23 |
| rlandy|ruck | quiquell|rover: yeah - had some fun with that yesterday | 12:23 |
| quiquell|rover | rlandy|ruck: and about test_volume_boot_pattern at f28 | 12:23 |
| sshnaidm | weshay, you can try, but not sure it works as in upstream.. | 12:23 |
| quiquell|rover | rlandy|ruck: looks like there is one container not starting it up | 12:23 |
| weshay | sshnaidm is that the ovb clean issue in post_failure? | 12:23 |
| quiquell|rover | rlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/1824977 | 12:23 |
| openstack | Launchpad bug 1824977 in tripleo "fedora-28 standalone failing at neutron-haproxy-ovnmeta service" [Critical,Triaged] | 12:23 |
| rlandy|ruck | arxcruz: btw: don;t think you were at yesterday's scrum - your hw is back | 12:23 |
| quiquell|rover | rlandy|ruck: also have run fs039 at reproducer and it's working fine, it resolves alright | 12:24 |
| rlandy|ruck | see the spreadsheet for new ip and hostname | 12:24 |
| quiquell|rover | rlandy|ruck: kforde is helping | 12:24 |
| rlandy|ruck | quiquell|rover: cool - what was the issue? | 12:24 |
| weshay | rlandy|ruck quiquell|rover pass me a link to the ticket please | 12:24 |
| rlandy|ruck | quiquell|rover; we had some load issues yesterday | 12:25 |
| quiquell|rover | weshay: I haven't create a ticket yet jus talking with kforde he want's to be at a living freeipa server | 12:25 |
| quiquell|rover | rlandy|ruck: haven't discover any issue of anything today :-((( | 12:25 |
| rlandy|ruck | quiquell|rover: merged the change for queens - hope that gets fixed | 12:25 |
| rlandy|ruck | quiquell|rover: rocky - finally :) | 12:26 |
| quiquell|rover | rlandy|ruck: well centosci was geting stuck, ykarel has re-launch | 12:26 |
| quiquell|rover | rlandy|ruck: so we still don't know, let me check | 12:26 |
| rlandy|ruck | board is green | 12:26 |
| rlandy|ruck | promoter ran yesterday | 12:26 |
| weshay | sshnaidm check it out https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 | 12:26 |
| weshay | should it vote? | 12:27 |
| quiquell|rover | rlandy|ruck: yep we are not bad | 12:27 |
| quiquell|rover | rlandy|ruck: but f28 is screw | 12:27 |
| quiquell|rover | also centosci queens/pike promotions are geting rusty | 12:28 |
| rlandy|ruck | quiquell|rover: by tempest or promotion? | 12:28 |
| rlandy|ruck | totally | 12:28 |
| rlandy|ruck | hopefully merged fix will run and pass there | 12:28 |
| quiquell|rover | rlandy|ruck: about f28 https://bugs.launchpad.net/tripleo/+bug/1824977 | 12:29 |
| openstack | Launchpad bug 1824977 in tripleo "fedora-28 standalone failing at neutron-haproxy-ovnmeta service" [Critical,Triaged] | 12:29 |
| rlandy|ruck | quiquell|rover: could you vote here pls: https://review.openstack.org/#/c/652078/ | 12:29 |
| rlandy|ruck | fs001 failure i think is unrelated | 12:29 |
| *** amoralej|lunch is now known as amoralej | 12:30 | |
| quiquell|rover | rlandy|ruck: I am going to workflow it | 12:30 |
| rlandy|ruck | quiquell|rover: k - ping me about that one before you log off if I should continue there | 12:30 |
| rlandy|ruck | thank you | 12:30 |
| quiquell|rover | rlandy|ruck: ack I am running a reproducer to see what's there | 12:31 |
| quiquell|rover | rlandy|ruck: Tengu is adding more logging there https://review.openstack.org/#/c/652978/ | 12:31 |
| Tengu | :) | 12:31 |
| Tengu | logging is life | 12:31 |
| Tengu | we love logging | 12:31 |
| rlandy|ruck | great - thanks | 12:31 |
| quiquell|rover | is "good" life | 12:31 |
| rlandy|ruck | marios: hey ... re: your comment on https://review.openstack.org/#/c/651247/ | 12:32 |
| rlandy|ruck | ok - to leave the rocky job in standalone all if it is non-voting? | 12:32 |
| rlandy|ruck | just to watch the run | 12:32 |
| *** jbadiapa has quit IRC | 12:32 | |
| marios | rlandy|ruck: yes but it is currently voting https://review.openstack.org/#/c/651247/ and only in check not gate | 12:34 |
| quiquell|rover | Tengu: I have a running system with the failing container | 12:34 |
| marios | rlandy|ruck: looking again | 12:34 |
| rlandy|ruck | mario: acking - adding that | 12:34 |
| marios | rlandy|ruck: so add into gate too https://review.openstack.org/#/c/651247/2/zuul.d/standalone-jobs.yaml | 12:34 |
| marios | rlandy|ruck: ack | 12:35 |
| quiquell|rover | Tengu: is there any way I can get the logs from a death container ? | 12:35 |
| Tengu | quiquell|rover: if the container is still there: podman logs <container> | 12:35 |
| Tengu | quiquell|rover: does it answer your need? | 12:37 |
| quiquell|rover | Tengu: I see something weird | 12:37 |
| Tengu | "I see dead people" | 12:37 |
| weshay | zbr quiquell|rover did you guys get hubbot turned off for patches? | 12:37 |
| rlandy|ruck | marios: actually - I think we should add it non-voting ( as I thought you suggested) first and watch it - if it's stable, great | 12:37 |
| Tengu | sorry :][ | 12:37 |
| marios | rlandy|ruck: acki mean that was my comment "decide and do either way" | 12:38 |
| weshay | quiquell|rover please abandon https://review.openstack.org/#/c/567224/ https://review.openstack.org/#/c/560445/ | 12:38 |
| zbr | weshay: nope, i forgot. let me try to look now. | 12:38 |
| quiquell|rover | weshay: they are abanodned | 12:39 |
| quiquell|rover | Tengu: https://paste.fedoraproject.org/paste/j-uLf92A9A~0Px3XpVd1ag | 12:40 |
| quiquell|rover | Tengu: looks like the service is trying to startup | 12:40 |
| quiquell|rover | But it takes so little time than I am not able to get the logs | 12:40 |
| Tengu | quiquell|rover: damn.... | 12:40 |
| Tengu | yeah, so yeah. you might find logs in neutron container, since that haproxy container is launched by neutron | 12:41 |
| Tengu | that's the best solution I have for now.... | 12:41 |
| Tengu | until my patch adding new logs merges, of course :) | 12:41 |
| quiquell|rover | Tengu: you mean inside the container ? | 12:43 |
| Tengu | quiquell|rover: one of the neutron services is trying to get that haproxy container up'n'running - I don't know which one though. But you should get logs for that neutron service and it should give you some reasons about the failure. | 12:44 |
| *** aakarsh has joined #oooq | 12:44 | |
| rlandy|ruck | quiquell|rover: weshay:so we want to bring anything up at the tripleo meeting? | 12:45 |
| rlandy|ruck | do we | 12:45 |
| weshay | as ruck / rover? | 12:45 |
| weshay | for bm | 12:45 |
| weshay | what context please | 12:45 |
| rlandy|ruck | weshay: ruck/rover | 12:46 |
| weshay | naw | 12:46 |
| rlandy|ruck | ok | 12:48 |
| *** mjturek has joined #oooq | 12:58 | |
| weshay | sshnaidm 1-1 | 13:01 |
| quiquell|rover | Tengu: I don't find anything at neutron containers logs | 13:03 |
| quiquell|rover | :-/ | 13:03 |
| Tengu | quiquell|rover: sooo... no other idea. You'll need to wait for the new logging thing to be merged and packaged :/ | 13:05 |
| quiquell|rover | ack | 13:05 |
| Tengu | quiquell|rover: hmm | 13:06 |
| Tengu | quiquell|rover: wait. | 13:06 |
| Tengu | quiquell|rover: if you have a reproducer, you might apply my patch locally, and restart the VM - that way, the newly launched container should get proper logging. | 13:06 |
| Tengu | hopefully the issue will stay after the reboot, of course. | 13:06 |
| quiquell|rover | Tengu: will try now I am back to testing freeipa issue | 13:07 |
| quiquell|rover | Tengu: thanks mate | 13:07 |
| Tengu | np - ping me if you want some help. Even on freeIPA, I have some experience with that beast (managing my own 3-node cluster on the side) | 13:07 |
| quiquell|rover | ohhh good to know | 13:08 |
| quiquell|rover | Tengu: so we are experiencing this issue https://bugs.launchpad.net/tripleo/+bug/1824772 | 13:09 |
| openstack | Launchpad bug 1824772 in tripleo "freeipa not resolving mirror.regionone.rdo-cloud-tripleo.rdoproject.org broken trust chain resolving" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 13:09 |
| quiquell|rover | Tengu: but with the reproducer is working fine | 13:09 |
| *** vinaykns has joined #oooq | 13:09 | |
| quiquell|rover | Tengu: could be infra though | 13:09 |
| rfolco | sshnaidm, do you understand why previous build failed? can we enable debug/verbose ? https://sf.hosted.upshift.rdu2.redhat.com/logs/29/167929/6/check/tripleo-ci-rhel-7-standalone-master-buildimage/7b5f4d8/logs/undercloud/home/zuul/overcloud_image_build.log.txt.gz | 13:09 |
| quiquell|rover | rlandy|ruck, ykarel: phase1 pike promoted | 13:13 |
| ykarel | quiquell|rover, cool | 13:13 |
| ykarel | and queens? | 13:13 |
| quiquell|rover | let's see queens | 13:13 |
| rlandy|ruck | woohoo | 13:13 |
| ykarel | okk | 13:13 |
| quiquell|rover | queens is green too | 13:13 |
| quiquell|rover | https://ci.centos.org/job/tripleo-quickstart-promote-queens-rdo_trunk-minimal/ | 13:14 |
| Tengu | quiquell|rover: keeping that tab open - I'm jumping in a couple of meetings, will check if I can help after that :) | 13:14 |
| quiquell|rover | Tengu: ack, thanks! | 13:14 |
| weshay | sshnaidm https://review.rdoproject.org/r/#/c/20239/ | 13:16 |
| quiquell|rover | weshay: this is no longer happening ? https://bugs.launchpad.net/tripleo/+bug/1822916 | 13:16 |
| openstack | Launchpad bug 1822916 in tripleo "tempest config error in standalone upgrade in reproducer only: TypeError: must be string or buffer, not callable-iterator" [High,Triaged] - Assigned to Quique Llorente (quiquell) | 13:16 |
| *** ccamacho has joined #oooq | 13:20 | |
| *** Goneri has joined #oooq | 13:20 | |
| *** dtantsur|brb is now known as dtantsur | 13:24 | |
| *** mjturek has quit IRC | 13:27 | |
| weshay | quiquell|rover can you join the community call for a few min | 13:33 |
| quiquell|rover | sure | 13:36 |
| *** Vorrtex has joined #oooq | 13:50 | |
| Tengu | quiquell|rover: might be linked? https://bugzilla.redhat.com/show_bug.cgi?id=577639 | 13:59 |
| openstack | bugzilla.redhat.com bug 577639 in bind "bind Stopped Resolving (broken trust chain resolving)" [Medium,Closed: notabug] - Assigned to atkac | 13:59 |
| Tengu | quiquell|rover: so maybe some forwarder failing to do their job? | 14:00 |
| quiquell|rover | Tengu: yep old friend this bug have read it for some days | 14:00 |
| Tengu | i.e. broken dnssec... | 14:00 |
| *** kopecmartin is now known as kopecmartin|off | 14:02 | |
| rlandy|ruck | weshay: quiquell|rover: https://bugs.launchpad.net/tripleo/+bug/1824772 | 14:03 |
| openstack | Launchpad bug 1824772 in tripleo "freeipa not resolving mirror.regionone.rdo-cloud-tripleo.rdoproject.org broken trust chain resolving" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 14:03 |
| marios | weshay: https://docs.google.com/presentation/d/1mNOD-4Vh1hi72mmWpyIMNQ5cmSRSQ8vIXDZa6g1kOkY/edit?usp=sharing | 14:04 |
| weshay | sshnaidm https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 | 14:07 |
| *** jbadiapa has joined #oooq | 14:15 | |
| *** dtantsur has quit IRC | 14:27 | |
| *** dtantsur has joined #oooq | 14:32 | |
| rlandy|ruck | rfolco: pls help me out here ... | 14:35 |
| rfolco | rlandy|ruck, whats up | 14:36 |
| rlandy|ruck | rfolco: did we have a bug for the failing buildah | 14:36 |
| rlandy|ruck | I can't find it | 14:36 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1822888? | 14:36 |
| openstack | Launchpad bug 1822888 in tripleo "f28 container build broken by tripleo-repos InvalidArguments('Branches only suppported with centos7')" [Critical,Fix released] | 14:36 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1823173 | 14:37 |
| openstack | Launchpad bug 1823173 in tripleo "images.rdoproject.org/master/rdo_trunk/current-tripleo images are missing" [Critical,Fix released] - Assigned to Quique Llorente (quiquell) | 14:37 |
| rlandy|ruck | ^^that one | 14:37 |
| rfolco | https://bugs.launchpad.net/tripleo/+bug/1824388 is the one you filed | 14:37 |
| openstack | Launchpad bug 1824388 in tripleo "periodic jobs are failing undercloud install - Not found image" [Critical,Fix released] - Assigned to Ronelle Landy (rlandy) | 14:37 |
| rlandy|ruck | idk | 14:37 |
| rlandy|ruck | rfolco: thanks | 14:38 |
| rfolco | rlandy|ruck, there is also one that emilien commented, you have it ? | 14:38 |
| rfolco | rlandy|ruck, the one I opened sometime ago | 14:38 |
| rfolco | rlandy|ruck, this https://bugs.launchpad.net/tripleo/+bug/1819632 | 14:39 |
| openstack | Launchpad bug 1819632 in tripleo "[gate] Error pulling image: invalid character '<' parsing 404 response" [Medium,Triaged] - Assigned to Emilien Macchi (emilienm) | 14:39 |
| rfolco | rlandy|ruck, there is probably much of overlap in these bugs | 14:40 |
| rlandy|ruck | rfolco++ | 14:40 |
| *** quiquell|rover is now known as quiquell|off | 14:41 | |
| *** mjturek has joined #oooq | 14:41 | |
| zbr | @oooq who is curious to hear bit more about molecule use? (very similar to what I did with release-config already). | 14:42 |
| weshay | rfolco what is your bluejeans # | 14:44 |
| rfolco | sec | 14:45 |
| rfolco | weshay, https://bluejeans.com/5878458097 | 14:45 |
| weshay | thanks | 14:46 |
| rfolco | weshay, always make confusion with yours... which ends with 98 | 14:46 |
| rfolco | :) | 14:46 |
| rfolco | weshay, thank you | 14:46 |
| rfolco | weshay, I am running push-true on local registry, now will test podman pull on these built containers by adding new post playbook here https://review.openstack.org/#/c/652126/ | 14:48 |
| rfolco | weshay, if you get any other ideas let me know | 14:48 |
| weshay | panda http://git.openstack.org/cgit/openstack/networking-ovn/tree/zuul.d/project.yaml#n30 | 14:50 |
| weshay | http://git.openstack.org/cgit/openstack/networking-ovn/tree/zuul.d/networking-ovn-jobs.yaml#n170 | 14:51 |
| panda | weshay: yep ... so hes' requesting a multinode job ... maybe we should suggest a standalone ... | 14:52 |
| *** ykarel is now known as ykarel|away | 14:57 | |
| *** ykarel|away is now known as ykarel | 15:22 | |
| *** hamzy has quit IRC | 15:22 | |
| *** holser_ has quit IRC | 15:25 | |
| *** chem has quit IRC | 15:26 | |
| chandankumar | rlandy|ruck: https://logs.rdoproject.org/24/639324/20/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/489537a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-04-16_13_17_21 | 15:28 |
| weshay | rlandy|ruck can we define a rocky standalone on internal sf as well ? | 15:28 |
| chandankumar | failure on fs039 during overcloud deploy is it a known issue? | 15:28 |
| weshay | marios ^ | 15:28 |
| ykarel | rfolco, fyi in case you not knew already, i filed a bug against buildah container build job today:- https://bugs.launchpad.net/tripleo/+bug/1824952 | 15:29 |
| openstack | Launchpad bug 1824952 in tripleo "Tempest Container Image is not building successfully using buildah" [High,Triaged] | 15:29 |
| marios | weshay: ack i can look into that tomorrow cc rlandy|ruck maybe we should add another task into that OSP14 story | 15:29 |
| weshay | aye | 15:30 |
| weshay | thanks | 15:30 |
| ykarel | rfolco, you should hit the same during your testing of push registry | 15:30 |
| rlandy|ruck | weshay: ack - will do | 15:30 |
| * marios shutdown sequence | 15:30 | |
| rfolco | ykarel, is it just tempest container ? | 15:30 |
| ykarel | rfolco, yes | 15:31 |
| rlandy|ruck | chandankumar; fs039 is a disaster atm | 15:31 |
| rfolco | ok, thanks for letting me know ykarel | 15:31 |
| rlandy|ruck | chandankumar: lots of freeipa issues | 15:31 |
| ykarel | rfolco, tempest only has that combo: WORKDIR + symlink | 15:31 |
| rfolco | rlandy|ruck, you may like to check this ^ | 15:31 |
| rfolco | yet another buildah issue | 15:32 |
| rlandy|ruck | chandankumar: https://bugs.launchpad.net/tripleo/+bug/1824772 - so short answer - yes, known issue | 15:32 |
| openstack | Launchpad bug 1824772 in tripleo "freeipa not resolving mirror.regionone.rdo-cloud-tripleo.rdoproject.org broken trust chain resolving" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 15:32 |
| rfolco | ykarel, I am trying to exercise push to local registry + pull with podman to see if we reproduce same issue we saw in promotion pipeline | 15:32 |
| ykarel | rfolco, yes good start ^^ | 15:33 |
| rlandy|ruck | ykarel: rfolco: thanks | 15:33 |
| ykarel | rfolco, try running both inspect and pull | 15:33 |
| rfolco | ykarel, ok, but skopeo inspect works for the containers we cannot access in rdo registry | 15:34 |
| rlandy|ruck | ykarel: going to merge https://review.rdoproject.org/r/#/c/20242/ and try reduced periodic load - pls comment if you have any objections | 15:34 |
| ykarel | rlandy|ruck, looking | 15:34 |
| weshay | crap another spike it failed heat stacks | 15:35 |
| weshay | looks ok now | 15:35 |
| chandankumar | rlandy|ruck: thanks, I will focus on other stuff currently, will wait till it gets resolved! | 15:35 |
| rlandy|ruck | comes down | 15:35 |
| rlandy|ruck | chandankumar: we have moved fs039 to non-voting | 15:36 |
| chandankumar | debugging this podman issue leads us to on how to read gocode | 15:36 |
| chandankumar | experience was kind of horrible | 15:36 |
| rlandy|ruck | lol - does not bode well | 15:37 |
| ykarel | rlandy|ruck, timings looks wrong | 15:39 |
| ykarel | 0,12,18 | 15:39 |
| ykarel | okk if pipleine finishes < 6 hours | 15:40 |
| rlandy|ruck | ykarel: I am trying to avoid the time when the 24 hr pipeline kicks | 15:40 |
| ykarel | which is true i think, | 15:40 |
| rlandy|ruck | most tests should be done but not all maybe | 15:40 |
| rfolco | daily starts at 4 | 15:40 |
| ykarel | 24hr is low priority | 15:41 |
| ykarel | so mainly waits if other jobs are running | 15:41 |
| rlandy|ruck | so there are two sechedules ... | 15:41 |
| rlandy|ruck | - time: '10 0,12,18 * * *' | 15:41 |
| rlandy|ruck | and | 15:41 |
| rlandy|ruck | - time: '10 9,15,21 * * *' | 15:41 |
| rlandy|ruck | trying to avoid 0 4 | 15:41 |
| rlandy|ruck | ykarel: what would you suggest? | 15:42 |
| ykarel | rlandy|ruck, ^^ should work i only doubted at pipeline run < 6 hours or not | 15:42 |
| rlandy|ruck | ykarel: honestly idk | 15:43 |
| rlandy|ruck | I think mostly it's done | 15:43 |
| rlandy|ruck | minis fs021 | 15:43 |
| rlandy|ruck | minus | 15:43 |
| rlandy|ruck | and fs020 | 15:43 |
| ykarel | hmm looks like | 15:43 |
| rlandy|ruck | long tempest | 15:43 |
| rlandy|ruck | ykarel: we can revert if I made the situation worse | 15:43 |
| ykarel | rlandy|ruck, ok let's try | 15:44 |
| rlandy|ruck | dlrn runs continuously so I think that will be ok | 15:44 |
| ykarel | it will improve i think | 15:44 |
| rlandy|ruck | let's hope :) | 15:44 |
| ykarel | +2 | 15:44 |
| ykarel | rlandy|ruck, also long gaps = more commits | 15:45 |
| rlandy|ruck | true | 15:45 |
| ykarel | so more commits to test, and more commits to compare to debug the issue | 15:45 |
| rlandy|ruck | we'll have to see what shakes out | 15:46 |
| ykarel | and current and last release gets more commits | 15:46 |
| ykarel | okk | 15:46 |
| rlandy|ruck | other option is to keep master on its schedule and reduce stein | 15:46 |
| *** marios has quit IRC | 15:50 | |
| ykarel | rlandy|ruck, so it was concluded the introspection issues were due to load on rdo-cloud? | 15:51 |
| rlandy|ruck | ykarel: they definitely increase when rdocloud is loaded | 15:51 |
| weshay | panda rfolco we need a user story for next sprint that creates a mini-pipeline for f28 only that uses buildah. Doesn't need to vote and should use a known good hash to start.. like previous-current-tripleo | 15:51 |
| ykarel | rlandy|ruck, okk | 15:52 |
| ykarel | so exact issue is still not clear | 15:52 |
| rlandy|ruck | it never is | 15:53 |
| chandankumar | ykarel: rlandy|ruck is there any plan to move fs21 like jobs to a seperate pipeline ? | 15:53 |
| chandankumar | as it takes longer hours | 15:53 |
| rlandy|ruck | chandankumar: I wasn't aware of any plans/discussion around this | 15:54 |
| rlandy|ruck | they do take longer | 15:54 |
| chandankumar | rlandy|ruck: just an idea | 15:54 |
| rfolco | weshay, https://tree.taiga.io/project/tripleo-ci-board/us/1025 | 15:55 |
| rlandy|ruck | chandankumar: it's decent idea - let's see if with the current change in trigger schedule, fs021 runs too long. If so, I'll propose a changed pipeline | 15:55 |
| rlandy|ruck | it would be tricky to get the timing right | 15:55 |
| rlandy|ruck | to get the right build hash | 15:55 |
| rlandy|ruck | and images | 15:55 |
| rlandy|ruck | marios: weshay:adding a standalone rocky job definition in internal sf is easy - question is how and when you want to trigger it | 16:00 |
| *** ykarel is now known as ykarel|away | 16:00 | |
| weshay | rlandy|ruck the internal osp-14/rocky could just be daily until we get more details | 16:01 |
| rlandy|ruck | weshay: k - can add that | 16:01 |
| *** skramaja has quit IRC | 16:02 | |
| *** ykarel|away has quit IRC | 16:07 | |
| *** bogdando has quit IRC | 16:15 | |
| sshnaidm | rfolco, the one before last job built centos images successfully: https://code.engineering.redhat.com/gerrit/#/c/167929/ here: https://sf.hosted.upshift.rdu2.redhat.com/logs/29/167929/6/check/tripleo-ci-rhel-7-standalone-master-buildimage/1dbc001/ | 16:17 |
| sshnaidm | rfolco, need now to tweak it to build rhel, added templates here: https://review.openstack.org/#/c/652720/9/playbooks/multinode-standalone.yml#63 | 16:18 |
| rfolco | nice sshnaidm | 16:18 |
| sshnaidm | rfolco, and it fails | 16:18 |
| sshnaidm | rfolco, passing centos build: https://sf.hosted.upshift.rdu2.redhat.com/logs/29/167929/6/check/tripleo-ci-rhel-7-standalone-master-buildimage/1dbc001/logs/undercloud/home/zuul/overcloud_image_build.log.txt.gz | 16:19 |
| sshnaidm | rfolco, failing rhel build: https://sf.hosted.upshift.rdu2.redhat.com/logs/29/167929/6/check/tripleo-ci-rhel-7-standalone-master-buildimage/fbbcc92/logs/undercloud/home/zuul/overcloud_image_build.log.txt.gz | 16:20 |
| rfolco | sshnaidm, I don't quite understand whats going on there, do you know what we need to do ? | 16:22 |
| sshnaidm | rfolco, not really, need to investigate | 16:28 |
| sshnaidm | rfolco, better to find someone on #tripleo to help | 16:28 |
| rfolco | sshnaidm, debug mode or verbose ? | 16:28 |
| sshnaidm | rfolco, this is in debug mode | 16:28 |
| rlandy|ruck | weshay: can we move this card to done: https://trello.com/c/yNNAaSqC/948-cixlp1824317tripleociproa-periodic-containers-build-fail-at-push-unauthorized-authentication-required-n? Left a comment about the plan going forward for testing buildah | 16:29 |
| rfolco | sshnaidm, aaah | 16:30 |
| rfolco | tmpfs | 16:30 |
| rfolco | 2019-04-16 14:27:49 | 2019-04-16 14:27:49.924 | + diskimage_builder/lib/common-functions:tmpfs_check:29 : RAM_NEEDED=14 | 16:30 |
| rfolco | 2019-04-16 14:27:49 | 2019-04-16 14:27:49.928 | + diskimage_builder/lib/common-functions:tmpfs_check:30 : '[' 8008244 -lt 14680064 ']' | 16:30 |
| sshnaidm | rfolco, it's the same in centos job | 16:30 |
| sshnaidm | rfolco, I think this is a problem: No source for a base image file configured. | 16:30 |
| *** ykarel|away has joined #oooq | 16:31 | |
| sshnaidm | rfolco, it's right before "trap_cleanup" | 16:31 |
| sshnaidm | rfolco, in centos it's: DIB_CLOUD_IMAGES=http://cloud.centos.org/centos/7/images | 16:32 |
| sshnaidm | DIB_CLOUD_IMAGES=http://cloud.centos.org/centos/7/images | 16:34 |
| sshnaidm | BASE_IMAGE_FILE=CentOS-7-x86_64-GenericCloud.qcow2.xz | 16:34 |
| sshnaidm | BASE_IMAGE_TAR=CentOS-7-x86_64-GenericCloud.qcow2.xz.tgz | 16:34 |
| sshnaidm | IMAGE_LOCATION=http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2.xz | 16:34 |
| sshnaidm | CACHED_IMAGE=/home/zuul/.cache/image-create/CentOS-7-x86_64-GenericCloud.qcow2.xz | 16:34 |
| sshnaidm | we should have this for rhel configured | 16:35 |
| sshnaidm | rfolco, but still, better to ask somebody that knows more.. | 16:35 |
| *** ccamacho has quit IRC | 16:53 | |
| *** yolanda_ has quit IRC | 16:57 | |
| zbr | rfolco: thanks for the comments on mo' review. | 16:58 |
| rfolco | zbr, looks good, getting close to get dumb people like me understand it :) | 16:59 |
| *** hamzy has joined #oooq | 17:27 | |
| *** amoralej is now known as amoralej|of | 17:29 | |
| *** amoralej|of is now known as amoralej|off | 17:29 | |
| *** dtantsur is now known as dtantsur|afk | 17:36 | |
| rlandy|ruck | weshay: we have a problem :( ... openstack-periodic-master should have kicked | 18:02 |
| weshay | hrm.. k.. at 2pm I take it | 18:03 |
| rlandy|ruck | no wait, I am wrong | 18:06 |
| rlandy|ruck | it trigger at 2:10 | 18:06 |
| rlandy|ruck | my mistake | 18:06 |
| rlandy|ruck | weshay: no we're ok - it just triggered | 18:10 |
| rlandy|ruck | panda++ | 18:14 |
| rlandy|ruck | you fixed the check issue on internal sf | 18:14 |
| weshay | rlandy|ruck if you need to move our 1-1 go ahead | 18:17 |
| weshay | actually we can knock it out today | 18:17 |
| rlandy|ruck | weshay: that time tomorrow should be fine - today works as well | 18:18 |
| * weshay avail when ever | 18:19 | |
| rlandy|ruck | weshay: ready to meet whenever you are | 18:25 |
| *** irclogbot_3 has quit IRC | 18:39 | |
| *** irclogbot_2 has joined #oooq | 18:40 | |
| weshay | rlandy|ruck k.. /me finishing up a thing.. ping you in 5 | 18:40 |
| rlandy|ruck | k | 18:40 |
| *** ykarel|away has quit IRC | 18:41 | |
| weshay | rlandy|ruck I'm in | 18:56 |
| rlandy|ruck | joining | 18:57 |
| rlandy|ruck | weshay: https://code.engineering.redhat.com/gerrit/#/c/167816/ | 19:03 |
| rlandy|ruck | weshay: https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset039.yml#L175 | 19:47 |
| rlandy|ruck | weshay: https://bugs.launchpad.net/tripleo/+bug/1821377 | 19:49 |
| openstack | Launchpad bug 1821377 in tripleo "TLS everywhere deployments fail when using many composable networks" [Medium,In progress] - Assigned to Harald Jensås (harald-jensas) | 19:49 |
| *** Vorrtex has quit IRC | 19:57 | |
| *** holser_ has joined #oooq | 20:13 | |
| weshay | #tripleo | 20:14 |
| weshay | CI Status: GREENISH, RDOCloud Status: YELLOW, promoted current-tripleo 3/18 | community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs/latest/ | 20:14 |
| weshay |  | 20:14 |
| weshay |  | 20:14 |
| weshay | SHOW OLDER MESSAGES | 20:14 |
| weshay | 10:06 | 20:14 |
| *** weshay has quit IRC | 20:14 | |
| *** weshay has joined #oooq | 20:15 | |
| rlandy|ruck | oh dear ... tripleo_common.image.exception.ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleomaster/fedora-binary-cinder-api:097206d3e2762d52939a593e2d7f7a671c722f96_3eee5076 | 20:15 |
| rlandy|ruck | weshay: can you connect to rdo registry | 20:20 |
| rlandy|ruck | tells me I am unauthorized all of a sudden | 20:20 |
| weshay | rlandy|ruck checking | 20:21 |
| weshay | it's taking it's sweet time | 20:21 |
| weshay | rlandy|ruck connected | 20:22 |
| weshay | brb | 20:22 |
| rlandy|ruck | weshay: weird - wanted to check this error: https://logs.rdoproject.org/openstack-periodic-master/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-standalone-master/64e95eb/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-04-16_19_55_41 - if the image/hash existed | 20:22 |
| rlandy|ruck | if it's the right hash | 20:23 |
| rlandy|ruck | envD master success - woohoo | 20:34 |
| weshay | rlandy|ruck nice | 20:37 |
| rlandy|ruck | weshay: question ... should these two be running concurrently? http://38.145.34.55/centos7_queens.log and http://38.145.34.55/centos7_rocky.log | 20:39 |
| weshay | hrm | 20:39 |
| weshay | I don't think so | 20:39 |
| rlandy|ruck | weshay: k - so we got two problems ... | 20:40 |
| rlandy|ruck | ^^ and fedora28 | 20:40 |
| weshay | hrm.. ya.. I see that | 20:41 |
| rlandy|ruck | and master ... http://38.145.34.55/centos7_master.log | 20:41 |
| weshay | trying to promote containers on both rocky and queens atm | 20:41 |
| rlandy|ruck | and master | 20:41 |
| rlandy|ruck | we're multi-tasking here | 20:41 |
| weshay | and master? | 20:41 |
| weshay | wth | 20:41 |
| rlandy|ruck | correct | 20:41 |
| rlandy|ruck | it's a party | 20:41 |
| weshay | oy | 20:41 |
| rlandy|ruck | let's get on promotion server | 20:42 |
| * rlandy|ruck opens tmate | 20:42 | |
| weshay | I am | 20:42 |
| weshay | ya | 20:42 |
| rlandy|ruck | oh | 20:42 |
| weshay | please | 20:42 |
| weshay | rlandy|ruck let's blue and tmate | 20:42 |
| rlandy|ruck | weshay: k- sent tmate | 20:43 |
| rlandy|ruck | joining blue | 20:43 |
| *** hamzy has quit IRC | 20:54 | |
| *** Goneri has quit IRC | 21:00 | |
| weshay | rfolco you changed the promoter service | 21:34 |
| rlandy|ruck | rfolco: you around? | 21:34 |
| weshay | rfolco need your assistance please | 21:34 |
| *** vinaykns has quit IRC | 21:37 | |
| *** Goneri has joined #oooq | 21:48 | |
| *** holser_ has quit IRC | 21:51 | |
| rlandy|ruck | weshay: rdocloud monitoring has stopped | 22:26 |
| rlandy|ruck | killed that container? | 22:26 |
| *** holser_ has joined #oooq | 22:36 | |
| *** Goneri has quit IRC | 22:50 | |
| *** holser_ has quit IRC | 22:53 | |
| *** Goneri has joined #oooq | 23:06 | |
| *** Goneri has quit IRC | 23:14 | |
| *** tosky has quit IRC | 23:19 | |
| *** rlandy|ruck is now known as rlandy|ruck|biab | 23:22 | |
| rlandy|ruck|biab | weshay: can you stop your script so we can re-enable rdocloud monitoring? | 23:23 |
| *** dsneddon has quit IRC | 23:25 | |
| *** dsneddon has joined #oooq | 23:27 | |
| *** dsneddon has quit IRC | 23:29 | |
| *** dsneddon has joined #oooq | 23:30 | |
| *** sshnaidm is now known as sshnaidm|afk | 23:50 | |
| *** rlandy|ruck|biab is now known as rlandy|ruck | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!