| *** dsneddon has joined #oooq | 00:04 | |
| *** dsneddon has quit IRC | 00:09 | |
| *** jmasud has quit IRC | 00:13 | |
| *** jmasud has joined #oooq | 00:14 | |
| *** dsneddon has joined #oooq | 00:36 | |
| *** dsneddon has quit IRC | 00:41 | |
| *** surajpatil1 has joined #oooq | 00:44 | |
| *** surajpatil1 has quit IRC | 00:53 | |
| *** tosky has quit IRC | 01:06 | |
| *** dsneddon has joined #oooq | 01:16 | |
| *** dsneddon has quit IRC | 01:21 | |
| *** dsneddon has joined #oooq | 01:50 | |
| *** dsneddon has quit IRC | 01:55 | |
| *** dsneddon has joined #oooq | 02:31 | |
| *** dsneddon has quit IRC | 02:36 | |
| *** dsneddon has joined #oooq | 03:12 | |
| *** dsneddon has quit IRC | 03:17 | |
| *** dsneddon has joined #oooq | 03:46 | |
| *** dsneddon has quit IRC | 03:51 | |
| *** ykarel|away is now known as ykarel | 03:52 | |
| *** skramaja has joined #oooq | 04:22 | |
| *** dsneddon has joined #oooq | 04:23 | |
| *** dsneddon has quit IRC | 04:28 | |
| *** sshnaidm|afk has quit IRC | 04:38 | |
| *** udesale has joined #oooq | 04:45 | |
| *** sshnaidm|afk has joined #oooq | 04:51 | |
| *** dsneddon has joined #oooq | 04:59 | |
| *** ysandeep|away is now known as ysandeep | 05:00 | |
| *** dsneddon has quit IRC | 05:03 | |
| *** udesale has quit IRC | 05:31 | |
| *** udesale has joined #oooq | 05:32 | |
| *** dsneddon has joined #oooq | 05:33 | |
| *** dsneddon has quit IRC | 05:38 | |
| *** raukadah is now known as chandankumar | 05:43 | |
| *** ratailor has joined #oooq | 05:53 | |
| *** dsneddon has joined #oooq | 06:10 | |
| *** dsneddon has quit IRC | 06:14 | |
| *** marios|ruck has joined #oooq | 06:14 | |
| *** udesale has quit IRC | 06:23 | |
| *** udesale has joined #oooq | 06:25 | |
| *** dsneddon has joined #oooq | 06:43 | |
| *** dsneddon has quit IRC | 06:48 | |
| *** dsneddon has joined #oooq | 07:22 | |
| *** jbadiapa has quit IRC | 07:23 | |
| *** dsneddon has quit IRC | 07:27 | |
| *** jbadiapa has joined #oooq | 07:28 | |
| *** dpawlik has joined #oooq | 07:29 | |
| *** ccamacho has joined #oooq | 07:33 | |
| *** sshnaidm|afk has quit IRC | 07:41 | |
| *** ykarel is now known as ykarel|lunch | 07:41 | |
| *** saneax has joined #oooq | 07:47 | |
| *** sshnaidm|afk has joined #oooq | 07:55 | |
| *** dsneddon has joined #oooq | 07:56 | |
| *** dsneddon has quit IRC | 08:01 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:04 | |
| sshnaidm | chandankumar, hi, do yo know maybe why 035 fails on ipv6 tempest here? https://review.opendev.org/#/c/708988/ I disabled it temporarily, it started to fail from Feb 19 | 08:09 |
|---|---|---|
| marios|ruck | sshnaidm: there is a bug for it | 08:11 |
| marios|ruck | sshnaidm: and fix sec (yatin fixed it) | 08:11 |
| marios|ruck | sshnaidm: but is blocked on new bug | 08:11 |
| marios|ruck | https://bugs.launchpad.net/tripleo/+bug/1864409 [master][fs035][ipv6] Deployments failing with 503 Service Unavailable(neutron) | 08:11 |
| openstack | Launchpad bug 1864409 in tripleo "[master][fs035][ipv6] Deployments failing with 503 Service Unavailable(neutron)" [Critical,In progress] - Assigned to yatin (yatinkarel) | 08:11 |
| marios|ruck | * https://review.opendev.org/#/c/709404/1 | 08:12 |
| marios|ruck | sshnaidm: any idea what's up with grafana btw weshay|ruck mentioned it on friday and looks pretty borked 503 bad gateways... didn't start investigating yet | 08:12 |
| *** tesseract has joined #oooq | 08:12 | |
| sshnaidm | marios|ruck, what's up with it? | 08:13 |
| sshnaidm | marios|ruck, doesn't it work for you? | 08:13 |
| marios|ruck | sshnaidm: i just refreshed and it is 'fixd' (though i refreshed earlier too)... did you do something ? | 08:14 |
| marios|ruck | sshnaidm: on friday weshay|ruck was pointing to it with same issue (like no data, and red in the corner with text '503 bad network gateway' or somesuch | 08:14 |
| sshnaidm | marios|ruck, I fixed only rdo jobs yesterday, they didn't appear there, that's it | 08:14 |
| sshnaidm | marios|ruck, mm, ok, didn't see that, maybe temporary glitch | 08:15 |
| marios|ruck | sshnaidm: yeah happened again just now. like when i refresh it... let me try close/new tab etc | 08:15 |
| sshnaidm | marios|ruck, it's http://dashboard-ci.tripleo.org, right? | 08:15 |
| marios|ruck | sshnaidm: yeah | 08:16 |
| marios|ruck | sshnaidm: same again ... like that https://i.imgur.com/RSIJ5Lj.png | 08:18 |
| marios|ruck | sshnaidm: i will try digging in a bit anyway sounds like the answer is 'no haven't seen it' | 08:18 |
| *** tosky has joined #oooq | 08:18 | |
| sshnaidm | marios|ruck, ok, see the problem now, out of memory.. | 08:19 |
| sshnaidm | marios|ruck, you can use the internal one meanwhile http://tripleo-cockpit.usersys.redhat.com | 08:19 |
| marios|ruck | sshnaidm: ack thanks | 08:20 |
| *** amoralej|off is now known as amoralej | 08:22 | |
| *** jfrancoa has joined #oooq | 08:25 | |
| *** ccamacho has quit IRC | 08:27 | |
| *** ccamacho has joined #oooq | 08:33 | |
| zbr | o/ | 08:35 |
| *** dsneddon has joined #oooq | 08:36 | |
| sshnaidm | marios|ruck, I'll take upstream grafana down for a maintenance | 08:40 |
| *** dsneddon has quit IRC | 08:41 | |
| marios|ruck | thank you sshnaidm s | 08:41 |
| *** ykarel|lunch is now known as ykarel | 08:46 | |
| *** udesale_ has joined #oooq | 08:49 | |
| *** udesale has quit IRC | 08:52 | |
| sshnaidm | zbr, https://raw.githubusercontent.com/thepracticaldev/orly-full-res/master/resolvingbrokendependencies-big.png | 08:59 |
| zbr | that is old news | 08:59 |
| *** bogdando has joined #oooq | 09:00 | |
| *** ccamacho has quit IRC | 09:01 | |
| zbr | you will see how many dependency overrides we will have to put to keep py27 env running. | 09:01 |
| zbr | sshnaidm: but the issue here is that tripleo-common has broken deps | 09:05 |
| ykarel | marios|ruck, hi | 09:05 |
| marios|ruck | ykarel: hey man thanks for the bug/fix for neutron thing | 09:06 |
| ykarel | marios|ruck, i ran fs035 master periodic with the fix https://review.rdoproject.org/r/#/c/25254/ | 09:06 |
| ykarel | it reported to dlrnapi and now master is promoting | 09:06 |
| marios|ruck | ykarel: nice :) thanks! | 09:06 |
| marios|ruck | ykarel++ | 09:07 |
| marios|ruck | ykarel: so i guess periodic pipeline isn't hitting that gitdb thing | 09:07 |
| ykarel | marios|ruck, not atleast in tripleo deploy jobs | 09:08 |
| ykarel | marios|ruck, i think the promotion due to depends-on unmerged tht patch should not effect anything | 09:08 |
| ykarel | upstream check fs035 should still fail even after promotion, as tht will be consumed from current repo | 09:09 |
| marios|ruck | ykarel: yeah until your fix merges you mean | 09:10 |
| marios|ruck | ykarel: its kind of 'cheating' promotino (also a bit dangerous, i mean incase we don't merge the depends-on then it means we promoted something that won't exist) | 09:10 |
| ykarel | marios|ruck, yes right | 09:10 |
| ykarel | marios|ruck, re. cheating if we do for non tripleo, then it's really bad | 09:11 |
| ykarel | but for tripleo ones it should not be that bad if we get the fixes land soon | 09:11 |
| marios|ruck | ykarel: i do it all the time, and in this case is fine i think but yeah just saying its somethign that was brough to my attention a while ago and we have to be sure about the depends-on | 09:11 |
| marios|ruck | k thanks ykarel | 09:11 |
| marios|ruck | ykarel: let's hope we get that gitdb fix today too it blocks all the things | 09:12 |
| ykarel | marios|ruck, it's in gate, so should merge soon | 09:12 |
| marios|ruck | ykarel: that? https://review.opendev.org/709389 | 09:12 |
| ykarel | marios|ruck, yes | 09:13 |
| marios|ruck | ykarel: /me refresh but 11:12 < ykarel> marios|ruck, it's in gate, so should merge soon | 09:13 |
| ykarel | mmm | 09:13 |
| marios|ruck | its not in the gate | 09:13 |
| *** dsneddon has joined #oooq | 09:13 | |
| ykarel | hmm i saw zuul +1 some time back and thought it moved to gate, but it's not | 09:13 |
| marios|ruck | ykarel: ack k yeah | 09:14 |
| ykarel | mmm no +W there :) | 09:14 |
| ykarel | so it's expected | 09:14 |
| zbr | does anyone know if python3 an pip3 executables are available on all platforms using py3? | 09:15 |
| *** dsneddon has quit IRC | 09:18 | |
| marios|ruck | docs review for ruck|rover if you have time thanks https://review.opendev.org/#/c/708914/ Document override tripleo-ci-testing with dlrn_hash_tag testproject | 09:19 |
| chandankumar | panda, Hello\ | 09:20 |
| chandankumar | panda, can you promote the current consistent hash to toci testing for centos-8? | 09:21 |
| panda | chandankumar: you mean all components in https://trunk.rdoproject.org/centos8-master/component/{}/consistent ? | 09:22 |
| panda | chandankumar: it depends, nowadays I charge 10EUR per promotion. | 09:22 |
| chandankumar | panda, yes | 09:22 |
| chandankumar | panda, I donot have EUROs, does rupees will work? | 09:23 |
| panda | chandankumar: I accept cash traveller's cheques, credit card debi card, bitcoins. | 09:23 |
| panda | chandankumar: promote to ? | 09:24 |
| chandankumar | panda, from here https://trunk.rdoproject.org/centos8-master/consistent/delorean.repo tripleo component to promoted component and then toci testing | 09:25 |
| panda | oh | 09:25 |
| chandankumar | panda, from here https://logserver.rdoproject.org/60/25260/1/check/periodic-tripleo-centos-8-master-component-compute-promote-to-promoted-components/1be62d6/job-output.txt I am need to add promotion critieria file, then it will be automatically done | 09:26 |
| panda | chandankumar: ok, that will take a bit, last manual promotion was a bit different than this | 09:27 |
| chandankumar | panda, I think I need the same clients also | 09:27 |
| chandankumar | https://logserver.rdoproject.org/97/24997/8/check/tripleo-centos-8-buildimage-overcloud-full/e168e1b/overcloud-full.log | 09:27 |
| panda | chandankumar: I have to extract some other part of the promoter code | 09:27 |
| panda | chandankumar: so that will be 20 EUR | 09:27 |
| chandankumar | panda, ok, send the details :-) | 09:28 |
| *** yolanda__ has joined #oooq | 09:37 | |
| *** apetrich has joined #oooq | 09:37 | |
| *** yolanda has quit IRC | 09:40 | |
| zbr | panda: i was thinking that if we would charge devs a fee per raised PR, we may be able to better select what matters | 09:44 |
| zbr | some kind of gamification on gerrit would be interesting | 09:45 |
| *** dsneddon has joined #oooq | 09:52 | |
| *** dsneddon has quit IRC | 09:57 | |
| panda | chandankumar: it will take a bit longer than expected, something is not working correctly, configparser refuses to identify the .repo fil as a INI file | 09:59 |
| chandankumar | panda, np :-) | 10:17 |
| chandankumar | sshnaidm, marios|ruck https://review.opendev.org/#/c/708599/ c8 image build ready for review | 10:17 |
| panda | chandankumar: promoting now to promoted-components | 10:21 |
| panda | a single promotion is taking a log .... | 10:22 |
| panda | lot* | 10:22 |
| panda | I'm gonna time it. | 10:22 |
| panda | chandankumar: promoting to tripleo-ci-testing new | 10:23 |
| chandankumar | panda, cool, thanks ! | 10:24 |
| marios|ruck | chandankumar: ack adding to reviews will check next run | 10:24 |
| sshnaidm | chandankumar, commented | 10:24 |
| panda | chandankumar: done, almost 2 minutes to promote an aggregate hash | 10:25 |
| zbr | panda: you own tripleo-ci-promotion-staging* jobs, right? | 10:26 |
| *** dsneddon has joined #oooq | 10:26 | |
| panda | zbr: no | 10:27 |
| panda | zbr: I know nothing | 10:27 |
| panda | zbr: I hear no evil | 10:27 |
| zbr | better to rename them using s/pro/com/ | 10:27 |
| panda | zbr: I speak no evil | 10:27 |
| zbr | panda: please help me on https://review.rdoproject.org/zuul/status/change/25246,17 | 10:28 |
| panda | o like the poeple on homeland says "nenti sacciu, nenti vitti" | 10:29 |
| panda | zbr: I wasn't there, and if I was there I was sleeping and dreaming of not being there at all | 10:29 |
| marios|ruck | non cero, e ma ci cero dormivo | 10:30 |
| marios|ruck | i knew we would get there eventually panda... i noticed you started extortion racket earlier ... finally the masks have fallen | 10:30 |
| panda | zbr: 2020-02-24 10:24:45.954274 | rdo-centos-7 | "msg": "Destination directory /home/zuul/promoter_venv/share/tripleo-common/container-images does not exist" | 10:31 |
| *** dsneddon has quit IRC | 10:31 | |
| panda | zbr: what did you change ? | 10:31 |
| panda | zbr: it's an error in log gathering not on the workflow | 10:31 |
| panda | zbr: the workflow flows workingly | 10:31 |
| panda | zbr: works flowingly | 10:32 |
| zbr | panda: take a look https://review.rdoproject.org/r/#/c/25246/ -- nothing that could produce this, afaik. | 10:32 |
| zbr | https://review.rdoproject.org/r/#/c/25246/17/test-requirements.txt was mandatory | 10:32 |
| zbr | i suspect they got borked at different time but they did not run due to tight filters | 10:33 |
| panda | zbr: also those job never got to be added to gating jobs | 10:35 |
| zbr | panda: should I make them nv? | 10:35 |
| panda | zbr: absolutely not, we should put them in gates too | 10:35 |
| zbr | panda: someone made linters and py27 jobs nv, over the weekend, my change was fixing these, which are clearly more important. | 10:36 |
| zbr | panda: lets experiment with https://review.rdoproject.org/r/#/c/25270/ and see if they are ok or not. | 10:40 |
| *** dsneddon has joined #oooq | 11:06 | |
| *** dsneddon has quit IRC | 11:11 | |
| chandankumar | panda, need the same for cloudops components | 11:17 |
| chandankumar | sorry ignore that | 11:18 |
| chandankumar | panda, you need to again promote consistent to promoted component to toci testing for clients components | 11:23 |
| *** udesale_ has quit IRC | 11:28 | |
| panda | chandankumar: why ? | 11:28 |
| chandankumar | panda, https://logserver.rdoproject.org/97/24997/8/check/tripleo-centos-8-buildimage-overcloud-full/7ff7fb3/overcloud-full.log | 11:28 |
| panda | chandankumar: I have the change ready now, but I 'd like to understand what you're changing | 11:28 |
| panda | chandankumar: so we have a new consistent repo ? | 11:28 |
| chandankumar | panda, gnocchi client packages are missing there | 11:28 |
| panda | ok | 11:28 |
| chandankumar | panda, I am working on adding standalone and build image jobs for c8 | 11:29 |
| chandankumar | panda, for few days, can we add some promotion critieria which will promote automatically? | 11:29 |
| panda | chandankumar: no, the code to do this has not been merged yet | 11:30 |
| panda | chandankumar: promotion launched, will take 5 minutes | 11:30 |
| chandankumar | panda, ok | 11:30 |
| panda | chandankumar: I need to discuss with jpen if we can make a bulk promotion | 11:30 |
| panda | 2 minuts is wuite sime times | 11:30 |
| *** ccamacho has joined #oooq | 11:31 | |
| chandankumar | panda, thanks! | 11:35 |
| sshnaidm | panda, do you know what is problem here? http://pastebin.test.redhat.com/838781 | 11:35 |
| sshnaidm | panda, it's deploy of new promoter server | 11:35 |
| sshnaidm | No handlers could be found for logger "promoter" | 11:36 |
| *** dsneddon has joined #oooq | 11:40 | |
| panda | sshnaidm: there's an error in the configuration, and the config object would like to print a debug message, but the error happens before the logging is configured, so you're seeing that error. | 11:40 |
| sshnaidm | panda, can you please fix it? | 11:41 |
| *** dsneddon has quit IRC | 11:44 | |
| panda | sshnaidm: not easy, it'a circular dependency, the config object needs logging to output error so it would need setup_logging to run first, but setup logging needs the log file to in the configuration to setting itself up properly. I will either need to extrac the log file from the config without the config object, or make the config object output logs on stout until logging is set_up | 11:47 |
| panda | sshnaidm: I'll try the second approach | 11:47 |
| sshnaidm | panda, thanks | 11:47 |
| sshnaidm | panda, also question, do we still need an external volume for mounting /var/lib/docker? | 11:48 |
| sshnaidm | panda, I don't think we'll need to keep it, since it contains transient containers anyway | 11:48 |
| sshnaidm | panda, I don't remember what was a reason behind it | 11:49 |
| panda | sshnaidm: because when the cleanup fail, and soemtimes it fails, if we dont' have the containers in a different partition the system will become unusable, it's difficult to log in, to debug, to reboot. The differeent paritition ensures that containers will disturb only a non essential partition. | 11:51 |
| sshnaidm | panda, ok, so for podman we'll need another mount for /var/lib/containers | 11:52 |
| panda | sshnaidm: wirking on the patch testing it locally.But until we fix the integration promotion jobs nothing can merge. It broke after the release of tripleo-common | 11:53 |
| panda | sshnaidm: a bind mount to the same partition should be sufficient | 11:53 |
| sshnaidm | panda, I can try it on new promoter server manually | 11:53 |
| panda | sshnaidm: is the new promoter centos8 ? | 11:53 |
| sshnaidm | panda, nope, 7 | 11:53 |
| panda | sshnaidm: testing manually in production is not enough, we have two main execution paths right now, soon we'll have three and lots more cases to test for errors. Manual will not cover everything in time. | 11:56 |
| *** ratailor has quit IRC | 11:57 | |
| sshnaidm | panda, is there a way to start promotion server there? I need it to run for rocky only | 11:58 |
| sshnaidm | s/server/service/ | 11:58 |
| sshnaidm | panda, it's in test mode still, so could be redeployed again when it's fixed finally | 11:59 |
| panda | sshnaidm: you can use add the --force-legacy parameter to dlrnapi-prooteter.sh, it will use the old code for promotion, it's skipp the new code entirely | 11:59 |
| sshnaidm | but need it now to run | 11:59 |
| panda | sshnaidm: to dlrn-promoter.py --force-legacy, inside dlrnapi_promoter.sh | 12:00 |
| sshnaidm | panda, I don't think it will be enough to tell we're good with testing | 12:00 |
| sshnaidm | panda, it should run with new code.. | 12:01 |
| sshnaidm | otherwise it doesn't prove a concept | 12:01 |
| sshnaidm | panda, any hack will be fine | 12:01 |
| panda | sshnaidm: I'll upload a new patch asap, then you can check it out on the new promoter. | 12:04 |
| sshnaidm | panda, ack, thanks | 12:04 |
| *** dsneddon has joined #oooq | 12:05 | |
| panda | sshnaidm: the fix will only fix the logging, the configuration there still contains an error. The logging help you see what the error is. Maybe DLRNAPI_PASSWORD is not exported. | 12:06 |
| sshnaidm | ack | 12:07 |
| *** dsneddon has quit IRC | 12:11 | |
| zbr | marios|ruck: chandankumar : https://review.rdoproject.org/r/#/c/25278/ - urgent. | 12:12 |
| sshnaidm | panda, seems like with last changes secret files are not copied to user directory: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/tenant_vars/tripleo-infra/servers.yml#L75-L89 | 12:15 |
| marios|ruck | chandankumar: thanks for checking https://review.opendev.org/#/c/708914/2/doc/source/ci/chasing_promotions.rst@194 | 12:16 |
| panda | sshnaidm: https://review.rdoproject.org/r/24684 changed a bit, but still not tested for your case. I'll add a test to check if config class can log now. | 12:16 |
| marios|ruck | chandankumar: let me know if you want me to add something more there 14:16 < marios|ruck> chandankumar: thanks for checking https://review.opendev.org/#/c/708914/2/doc/source/ci/chasing_promotions.rst@194 | 12:19 |
| marios|ruck | ah i see you replied thanks chandankumar | 12:19 |
| sshnaidm | panda, it started after I copied files manually, so let's wait for a proper fix then | 12:23 |
| chandankumar | zbr, please change the nodeset of linters https://review.rdoproject.org/r/#/c/25278/ here so py3 changes should work | 12:27 |
| zbr | chandankumar: we cannot do it only for linters. | 12:27 |
| zbr | we can change the nodeset for all tox jobs, which is a problem at the moment. | 12:27 |
| zbr | that is due to the way rdo jobs are inhereted, nodeset is defined only on base job, and that has huge implications. | 12:28 |
| zbr | we will do it, but it will take a lot of work. | 12:28 |
| chandankumar | ok | 12:28 |
| zbr | we could try to override nodeset on job by job use. | 12:29 |
| chandankumar | marios|ruck, https://review.rdoproject.org/r/#/c/24701/ | 12:40 |
| marios|ruck | chandankumar: ack adding to list for rechecking | 12:41 |
| zbr | marios|ruck: chandankumar please read my comment on https://bugs.launchpad.net/tripleo/+bug/1864399/comments/2 -- this a blocker, probably you want to create new bug for it. | 12:50 |
| openstack | Launchpad bug 1864399 in tripleo "ERROR: Could not find a version that satisfies the requirement gitdb2===4.0.1" [Critical,Triaged] | 12:50 |
| zbr | while the errors seem similar, they are caused by very different reasons. | 12:51 |
| zbr | pypi mirrors/indexes are clearly borked, try to browse them. | 12:51 |
| zbr | even if you correct the URL, you still endup with a "selective" set of packages, one that would broke most of our tox jobs. | 12:52 |
| *** rlandy has joined #oooq | 12:56 | |
| marios|ruck | zbr: ok thanks | 12:57 |
| *** beagles has joined #oooq | 12:59 | |
| marios|ruck | folks scrum | 13:01 |
| weshay|ruck | zbr, chandankumar sshnaidm panda rlandy scrub | 13:01 |
| marios|ruck | panda: zbr: rlandy: chandankumar: | 13:01 |
| weshay|ruck | zbr, | 13:02 |
| weshay|ruck | sshnaidm, | 13:02 |
| zbr | weshay|ruck: wait, meet SSO takes time, a lot of it. | 13:02 |
| *** marios|ruck is now known as marios|ruck|call | 13:03 | |
| *** rfolco has joined #oooq | 13:04 | |
| marios|ruck|call | 14:50 #oooq: <@zbr> marios|ruck: chandankumar please read my comment on https://bugs.launchpad.net/tripleo/+bug/1864399/comments/2 -- this a blocker, probably you want to create new bug for it. | 13:04 |
| openstack | Launchpad bug 1864399 in tripleo "ERROR: Could not find a version that satisfies the requirement gitdb2===4.0.1" [Critical,Triaged] | 13:04 |
| chandankumar | weshay|ruck, https://review.opendev.org/#/c/709437/ standlonw already in progress | 13:07 |
| weshay|ruck | https://hackmd.io/FyI1eRz9SoCdgnsITR3PuQ?both | 13:07 |
| *** amoralej is now known as amoralej|lunch | 13:28 | |
| *** skramaja has quit IRC | 13:29 | |
| chandankumar | rfolco, rlandy https://review.opendev.org/#/c/709437/ | 13:36 |
| chandankumar | rlandy, let's finish the test project run, then we can sync | 13:39 |
| rlandy | sure | 13:40 |
| rlandy | https://logserver.rdoproject.org/95/24995/17/check/periodic-tripleo-ci-centos-8-standalone-master/c7bcce4/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz | 13:57 |
| rlandy | close | 13:57 |
| rlandy | Cannot download ansible-role-container-registry-1.1.1-0.20191218091847.7a2a664.el8.noarch.rpm: All mirrors were tried | 13:57 |
| rlandy | chandankumar: ^^ new? | 13:57 |
| chandankumar | rlandy, it was same issue, we were seeing in morning | 13:58 |
| rlandy | chandankumar: should I abandon https://review.opendev.org/#/c/709215/? | 13:58 |
| chandankumar | rlandy, yes | 13:59 |
| rlandy | chandankumar: ok - so we're sitting with ansible-role-container-registry-1.1.1-0.20191218091847.7a2a664.el8.noarch.rpm: All mirrors were tried now | 13:59 |
| chandankumar | ykarel, https://trunk.rdoproject.org/centos8/component/tripleo/promoted-components/ it is here also | 13:59 |
| chandankumar | ykarel, may be got missed in tripleo-ci-testing | 14:00 |
| zbr | weshay|ruck: who can explain me how our pypi mirrors are working | 14:00 |
| ykarel | chandankumar, yes | 14:00 |
| *** marios|ruck|call is now known as marios|ruck | 14:00 | |
| rlandy | chandankumar: the two files are different | 14:00 |
| chandankumar | rlandy, that package is not available in toci testing repo | 14:00 |
| chandankumar | it is available in consistent and promoted component that's why it failed | 14:01 |
| * rlandy gets lost here | 14:03 | |
| weshay|ruck | zbr, infra | 14:03 |
| weshay|ruck | zbr, please go learn .. so we can a better understanding | 14:03 |
| rlandy | chandankumar: why is there no equivalent of lines 54075 from master.yaml in promotion-testing-hash-master.yml? | 14:04 |
| rlandy | 54-75 | 14:04 |
| *** saneax has quit IRC | 14:07 | |
| weshay|ruck | chandankumar, let's do a quick pm/ua | 14:10 |
| weshay|ruck | https://meet.google.com/esg-fqsx-mth?authuser=1 | 14:11 |
| chandankumar | weshay|ruck, https://review.rdoproject.org/zuul/builds?pipeline=openstack-promote-component%20 | 14:13 |
| chandankumar | weshay|ruck, https://logserver.rdoproject.org/60/25260/1/check/periodic-tripleo-centos-8-master-component-compute-promote-to-promoted-components/1be62d6/job-output.txt | 14:13 |
| chandankumar | weshay|ruck, "msg": "file not found: /home/zuul/src/review.rdoproject.org/rdo-infra/ci-config/ci-scripts/dlrnapi_promoter/config/CentOS-8/component/master.yaml" | 14:14 |
| weshay|ruck | panda, hey | 14:14 |
| weshay|ruck | please join.. rlandy maybe as well | 14:14 |
| rfolco | zbr, can't we make py27 nv until it gets fixed or really useless ? py36 is there and green | 14:15 |
| rfolco | zbr, what are the plans for py27? its blocking my patches in ci-config :( | 14:16 |
| zbr | rfolco: we cannot, it is there because it tests something. if it fails, almost for sure that that code will also fail in production. | 14:16 |
| zbr | rfolco: ping ruck/rovers about https://bugs.launchpad.net/tripleo/+bug/1864399/comments/2 | 14:16 |
| openstack | Launchpad bug 1864399 in tripleo "ERROR: Could not find a version that satisfies the requirement gitdb2===4.0.1" [Critical,Triaged] | 14:16 |
| zbr | the same bug is breaking many molecule jobs, so you will not progress anyway. | 14:16 |
| rfolco | marios|ruck, is there a fix in progress for this ^ ? | 14:16 |
| marios|ruck | rfolco: yeah merged little while ago | 14:17 |
| marios|ruck | rfolco: https://bugs.launchpad.net/tripleo/+bug/1864399/comments/1 | 14:17 |
| openstack | Launchpad bug 1864399 in tripleo "ERROR: Could not find a version that satisfies the requirement gitdb2===4.0.1" [Critical,Triaged] | 14:17 |
| marios|ruck | Should be fixed with https://review.opendev.org/#/c/709389 | 14:17 |
| chandankumar | rlandy, https://review.rdoproject.org/r/#/c/25260/1/zuul.d/projects.yaml | 14:18 |
| *** Goneri has joined #oooq | 14:19 | |
| zbr | weshay|ruck: i am almost sure that the breakage was caused by enablement of rdo mirrors. | 14:19 |
| chandankumar | weshay|ruck, https://logserver.rdoproject.org/openstack-promote-component/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-component-security-promote-to-promoted-components/f92fce5/job-output.txt run from pipe;ine | 14:20 |
| rlandy | chandankumar: https://review.rdoproject.org/r/#/c/25064/ | 14:22 |
| weshay|ruck | chandankumar, https://gist.github.com/weshayutin/c77a2aa33c0d7a4c9b93481dc2f789f4 | 14:23 |
| zbr | weshay|ruck: marios|ruck can we have a meet to address the mirrors? | 14:24 |
| zbr | i am looking at https://github.com/rdo-infra/rdo-jobs/blame/9c3bcf79d8490571cbf6423c203aae24418e0b4c/roles/mirror-info-fork/templates/mirror_info.sh.j2#L52 | 14:24 |
| zbr | and it does not make much sense to me | 14:24 |
| *** amoralej|lunch is now known as amoralej | 14:24 | |
| zbr | that part $AFS_SLUG in particular | 14:24 |
| weshay|ruck | rfolco, ping | 14:25 |
| weshay|ruck | https://meet.google.com/esg-fqsx-mth?authuser=1 | 14:26 |
| rfolco | weshay|ruck, joining | 14:26 |
| zbr | because this is what we have on the server: http://mirror.regionone.rdo-cloud.rdoproject.org/wheel/ | 14:26 |
| zbr | no 7.7, no 8, so clearly not something that will ever work for centos. | 14:26 |
| marios|ruck | zbr: sure but gimme few (not really looked at that mirror_info.sh.j2 before either fwiw) | 14:29 |
| marios|ruck | panda: do you think we have a new issue there? http://38.145.34.55/centos7_master.log | 14:43 |
| marios|ruck | panda: failed master promotion cc weshay|ruck | 14:43 |
| weshay|ruck | hrm | 14:43 |
| weshay|ruck | k | 14:43 |
| marios|ruck | doesn't look like the usual trace for a missing image but the hash shouldn't be old enough for pruning anyway this looks different | 14:44 |
| marios|ruck | weshay|ruck: panda: we'll see if the same trace happens with stein it is also currently trying to promote after master failed * http://38.145.34.55/centos7_stein.log * 2020-02-24 13:24:13,927 32759 INFO promoter Promoting the container images for dlrn hash dd8d8b52c7702852e93498cf7551667f05cc4568 on stein to current-tripleo | 14:46 |
| rlandy | chandankumar: weshay|ruck: I am rekicking https://review.rdoproject.org/r/#/c/25064/ to get a new tripleo-ci-testing hash and the associated containers | 14:47 |
| rlandy | chandankumar: ^^ that is what you wanted, right? | 14:47 |
| marios|ruck | sshnaidm: panda: weshay|ruck: anyone been working on promoter today Feb 24 11:44:47 promoter dlrn-promoter-service.sh: error: unable to unlink old 'ci-scripts/dlrnapi_promoter/config/CentOS-7/train.ini' (Operation not permitted) | 14:50 |
| marios|ruck | panda: that ok panda? sorry i am digging in promoter journal now trying to understand | 14:50 |
| marios|ruck | Feb 24 11:44:47 promoter dlrn-promoter-service.sh: fatal: Could not reset index file to revision 'origin/master'. | 14:50 |
| zbr | marios|ruck: i wonder wtf added gitdb2===4.0.1 to https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt while it does not exist on https://pypi.org/project/gitdb2/4.0.2/ | 14:51 |
| panda | marios|ruck: that is normal, weshay|ruck trick to stop repo update | 14:51 |
| sshnaidm | marios|ruck, not on production one.. | 14:51 |
| marios|ruck | panda: hmmm maybe one of the containers actually failed | 14:51 |
| marios|ruck | Feb 24 11:25:38 promoter dockerd: time="2020-02-24T11:25:38.929343439Z" level=info msg="failed to mount layer sha256:aba8c6d454c2c54eaca0d95dd65ba86424ba764cc46280a67ba4b046973e38df (sha256:38710dca8610d5b57964104f28bdd9229c3e12da77b7e005a646ea819ea4c713) from docker.io/tripleostein/centos-binary-skydive-analyzer: unknown blob" | 14:51 |
| marios|ruck | Feb 24 11:32:38 promoter dockerd: time="2020-02-24T11:32:38.908974545Z" level=error msg="Not continuing with push after error: context canceled" | 14:52 |
| panda | marios|ruck: so I see the client failing to get manifest for nova api, but nova-api was pushed correctly before | 14:52 |
| marios|ruck | panda: can't find something in http://38.145.34.55/container-push/20200224-132413.log (that's the right log, right?) | 14:52 |
| marios|ruck | k thanks sshnaidm | 14:52 |
| marios|ruck | zbr: sorry digging at this promoter issue but did you see commit message at the fix | 14:53 |
| marios|ruck | zbr: it might offer some clues (wasn't clear to the author about what versions go with what ) | 14:53 |
| marios|ruck | zbr: that one * https://review.opendev.org/#/c/709389/5 | 14:53 |
| panda | marios|ruck: in fact, the hash in nova api really is missing | 14:55 |
| panda | marios|ruck: https://hub.docker.com/r/tripleomaster/centos-binary-nova-api/tags | 14:55 |
| panda | marios|ruck: https://hub.docker.com/r/tripleomaster/centos-binary-nova-api/tags?page=1&ordering=last_updated _x64_64 missing | 14:56 |
| panda | marios|ruck: we have the others but not _x64_64 for nova api | 14:56 |
| marios|ruck | wtf how did that happen | 14:56 |
| marios|ruck | possibly transient issue but then why did it upload the rest but not nova api | 14:56 |
| panda | marios|ruck: way worse than that, it reported the image pushed | 14:57 |
| marios|ruck | panda: wtf | 14:58 |
| panda | TASK [containers-promote : Tag and push x86_64 container images to docker.io] ***n | 14:58 |
| panda | ok: [localhost] => (item=neutron-metadata-agent-ovn) | 14:58 |
| panda | ok: [localhost] => (item=nova-api) | 14:58 |
| panda | ok: [localhost] => (item=nova-compute-ironic) | 14:58 |
| panda | ok: [localhost] => (item=nova-comput)e | 14:58 |
| marios|ruck | panda: but where are you looking could that be an earlier iteration? | 14:58 |
| marios|ruck | panda: http://38.145.34.55/centos7_master.log | 14:58 |
| marios|ruck | i see what you mean | 14:58 |
| panda | marios|ruck: so either docker is lying, or that image doesn't have _x86_64 tag part in it | 14:58 |
| *** TrevorV has joined #oooq | 15:00 | |
| marios|ruck | panda: s[m@192 ~]$ skopeo inspect docker://docker.io/tripleomaster/centos-binary-nova-compute:da4c8b0acc02c322be90322cef0517f785b14b27_b5a8f808_x86_64 | 15:03 |
| marios|ruck | panda: there | 15:03 |
| marios|ruck | panda: but not api | 15:04 |
| marios|ruck | [m@192 ~]$ skopeo inspect docker://docker.io/tripleomaster/centos-binary-nova-api:da4c8b0acc02c322be90322cef0517f785b14b27_b5a8f808_x86_64 | 15:04 |
| marios|ruck | FATA[0002] Error parsing image name "docker://docker.io/tripleomaster/centos-binary-nova-api:da4c8b0acc02c322be90322cef0517f785b14b27_b5a8f808_x86_64": Error reading manifest da4c8b0acc02c322be90322cef0517f785b14b27_b5a8f808_x86_64 in docker.io/tripleomaster/centos-binary-nova-api: manifest unknown: manifest unknown | 15:04 |
| panda | marios|ruck: we need more debugging | 15:06 |
| marios|ruck | panda: can't find something in journal yet | 15:08 |
| panda | marios|ruck: I think we need to temprarily increate the verbosity of the ansible playbook | 15:09 |
| marios|ruck | panda: please go ahead ? you want to start a tmux or just go ahead np | 15:09 |
| panda | marios|ruck: something is silently faining there, and the current log doesn't shed any light | 15:09 |
| marios|ruck | panda: well wait | 15:09 |
| marios|ruck | panda: cos stein is currently doing containers | 15:09 |
| marios|ruck | panda: lets see if it completes first and then ? | 15:09 |
| panda | marios|ruck: ok, but the logging is still at default level. | 15:09 |
| panda | marios|ruck: ok | 15:09 |
| marios|ruck | panda: * 2020-02-24 13:24:13,927 32759 INFO promoter Promoting the container images for dlrn hash dd8d8b52c7702852e93498cf7551667f05cc4568 on stein to current-tripleo | 15:09 |
| *** ykarel is now known as ykarel|afk | 15:12 | |
| *** ykarel|afk is now known as ykarel|away | 15:15 | |
| chandankumar | rlandy, sorry headed home, looking now | 15:17 |
| chandankumar | rlandy, looks good, thanks! | 15:17 |
| rlandy | chandankumar: one question- what must the priorities be changed to? | 15:18 |
| rlandy | https://review.opendev.org/#/c/708968/ | 15:18 |
| rlandy | in ^^ | 15:18 |
| rlandy | see comment on3. | 15:18 |
| chandankumar | rlandy, setting the priority of those repos | 15:18 |
| rlandy | to what priority? | 15:19 |
| chandankumar | rlandy, delorean.repo and delorean-master-deps.repo and tripleo-components repo | 15:20 |
| chandankumar | rlandy, sorry | 15:21 |
| chandankumar | rlandy, https://review.opendev.org/#/c/708968/9..19/config/release/tripleo-ci/CentOS-8/master.yml | 15:21 |
| zbr | panda: rfolco: py27 fix https://review.rdoproject.org/r/#/c/25246/ | 15:21 |
| chandankumar | rlandy, we only need to set the priority of tripleo-component repo to 10 | 15:21 |
| chandankumar | as include packages repo has priority 10 | 15:22 |
| chandankumar | rest is ok | 15:22 |
| rlandy | chandankumar: why do you even need that? if you have "https://trunk.rdoproject.org/centos8-master/{{ dlrn_hash_tag }}/delorean.repo" | 15:22 |
| rlandy | it's not in tripleo-ci-testing one | 15:23 |
| rlandy | should it be? | 15:23 |
| chandankumar | rlandy, https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo -> current tripleo | 15:24 |
| chandankumar | and same is needed for tripleo-ci-testing | 15:25 |
| rlandy | hmmm - k - adding | 15:26 |
| chandankumar | rlandy, call? | 15:27 |
| rlandy | ok | 15:27 |
| *** marios|ruck is now known as marios|ruck|call | 15:27 | |
| chandankumar | rlandy, https://meet.google.com/eha-davn-jww?pli=1&authuser=0 | 15:28 |
| marios|ruck|call | zbr: gimme link please to review | 15:29 |
| weshay|ruck | back from mtgs | 15:37 |
| weshay|ruck | 0. | 15:37 |
| weshay|ruck | 0/ | 15:37 |
| weshay|ruck | chandankumar, everything get worked out to unblock you? | 15:37 |
| chandankumar | weshay|ruck, yes | 15:37 |
| weshay|ruck | cool | 15:38 |
| weshay|ruck | panda, marios|ruck|call if master is having a hard time promoting.. I suggest we turn off manifest push | 15:39 |
| weshay|ruck | rlandy, which release file are you working on now.. https://review.opendev.org/#/c/708968/20/config/release/tripleo-ci/CentOS-8/promotion-testing-hash-master.yml or https://review.opendev.org/#/c/708968/20/config/release/tripleo-ci/CentOS-8/master.yml | 15:40 |
| rlandy | weshay|ruck: both | 15:41 |
| rlandy | running two checks | 15:41 |
| weshay|ruck | zbr, if there are more patches required to get our promoter server check jobs running besides https://review.rdoproject.org/r/#/c/25246/ please pursue w/ vigor | 15:41 |
| weshay|ruck | rlandy, afaict.. neither is going to work | 15:42 |
| marios|ruck|call | weshay|ruck: k not clear yet though if its manifests | 15:42 |
| rlandy | weshay|ruck: probably - but in what why are you referring to? | 15:42 |
| rlandy | way | 15:42 |
| weshay|ruck | marios|ruck|call, ya.. experience tells me at least to get through the master promotion... disable / check df -h / restart | 15:42 |
| weshay|ruck | will get us through | 15:43 |
| weshay|ruck | rlandy, | 15:43 |
| weshay|ruck | - type: generic | 15:43 |
| weshay|ruck | reponame: delorean-current | 15:43 |
| weshay|ruck | filename: delorean-current.repo | 15:43 |
| weshay|ruck | baseurl: $RDO_DLRN_REPO | 15:43 |
| weshay|ruck | priority: 10 | 15:43 |
| weshay|ruck | includepkgs: | 15:43 |
| weshay|ruck | - ansible-role-container-registry | 15:43 |
| weshay|ruck | - ansible-role-tripleo* | 15:43 |
| weshay|ruck | - ansible-tripleo-ipsec | 15:43 |
| weshay|ruck | - instack | 15:43 |
| weshay|ruck | - instack-undercloud | 15:43 |
| weshay|ruck | - openstack-tripleo-* | 15:43 |
| weshay|ruck | - os-apply-config | 15:43 |
| marios|ruck|call | weshay|ruck: k we are waiting for stein to complete and panda was gonna increase verbosity too | 15:43 |
| weshay|ruck | - os-collect-config | 15:43 |
| weshay|ruck | - os-net-config | 15:43 |
| weshay|ruck | - os-refresh-config | 15:43 |
| weshay|ruck | - puppet-* | 15:43 |
| weshay|ruck | - python*-tripleo* | 15:43 |
| weshay|ruck | - python*-paunch* | 15:43 |
| weshay|ruck | - paunch-services | 15:43 |
| weshay|ruck | - tripleo-ansible | 15:43 |
| weshay|ruck | - ansible-config_template | 15:43 |
| marios|ruck|call | panda: sound ok? weshay|ruck suggests we disable manifests for next master push.. or we can try one with just debug and then another with manifest disabled? | 15:44 |
| weshay|ruck | the delorean-current repo file for centos8 has all the compoents | 15:44 |
| weshay|ruck | https://trunk.rdoproject.org/centos8-master/current/delorean.repo | 15:44 |
| weshay|ruck | rlandy, ^ that includes list should only be on the tripleo-component | 15:44 |
| weshay|ruck | rlandy, who are you working w/ on this? chandankumar ? | 15:44 |
| rlandy | ack | 15:44 |
| weshay|ruck | we should be able to construct something temporary to get us through | 15:44 |
| weshay|ruck | w/o it being perfect | 15:44 |
| rlandy | that is what we are trying to do | 15:44 |
| rlandy | get something to spring standalone | 15:45 |
| weshay|ruck | rlandy, chandankumar so I can put up a patch w/ what I think might work.. or we can chat ur choice | 15:45 |
| rlandy | weshay|ruck: nothing is in stone | 15:45 |
| rlandy | sure - go ahead - and we can discuss | 15:45 |
| weshay|ruck | aye.. only about getting it usuABLE | 15:45 |
| weshay|ruck | k | 15:45 |
| rlandy | ack | 15:46 |
| rlandy | perfect went out the window a long time ago | 15:46 |
| rlandy | we have added the components from current | 15:46 |
| rlandy | those packages can go on that | 15:46 |
| weshay|ruck | huh.. chandankumar rlandy did this recently change? https://trunk.rdoproject.org/centos8-master/component/tripleo/promoted-components/delorean.repo | 15:52 |
| weshay|ruck | was expecting to see all the components in this file.. ok.. this is good | 15:52 |
| rlandy | weshay|ruck: that's weird | 15:52 |
| panda | marios|ruck|call: if docker says it's pushing somthing that is not there in the end, I'm not sure how much this will work without moanifests. | 15:53 |
| rlandy | oh wait - that's right | 15:53 |
| rlandy | weshay|ruck: always been like that | 15:53 |
| *** marios|ruck|call is now known as marios|ruck | 15:54 | |
| panda | marios|ruck|call: but maybe there problem is hidden somewhere else, and skipping the manifest code will work.\ | 15:54 |
| panda | marios|ruck: ^ | 15:54 |
| marios|ruck | panda: ack ... lets see if the next one works and with increased verbosity | 15:54 |
| marios|ruck | panda: then we can try disable manifests | 15:54 |
| marios|ruck | panda: but lets post review to master.ini NOT MANUALLY please ? | 15:55 |
| marios|ruck | panda: i mean to manifest_push: false if we decide to | 15:55 |
| marios|ruck | weshay|ruck: that OK? 17:54 < marios|ruck> panda: ack ... lets see if the next one works and with increased verbosity | 15:55 |
| weshay|ruck | marios|ruck, panda k.. if it doesn't promote by the late afternoon, I'm going to update it.. k? | 15:56 |
| weshay|ruck | rlandy, chandankumar ok.. here are my thoughts in a review https://review.opendev.org/#/c/709560/1/config/release/tripleo-ci/CentOS-8/promotion-testing-hash-master.yml | 15:57 |
| marios|ruck | weshay|ruck: ack ok | 15:57 |
| weshay|ruck | rlandy, chandankumar eek.. one change | 15:57 |
| rlandy | weshay|ruck: you need the mirror definitions | 15:58 |
| weshay|ruck | rlandy, it will fail w/o in prmotion jobs? | 15:58 |
| weshay|ruck | OH I see sec | 15:58 |
| marios|ruck | panda: weshay|ruck: looks like stein completed 2020-02-24 14:54:21,279 32759 INFO promoter SUCCESS promoting centos7-stein tripleo-ci-testing as current-tripleo ({'timestamp': 1582516903, 'distro_hash': 'aae288b7e40e2f85dfa750a975c2060ffb484f75', 'promote_name': 'tripleo-ci-testing', 'user': 'review_rdoproject_org', 'repo_url': | 15:58 |
| marios|ruck | 'https://trunk.rdoproject.org/centos7-stein/dd/8d/dd8d8b52c7702852e93498cf7551667f05cc4568_aae288b7', 'full_hash': 'dd8d8b52c7702852e93498cf7551667f05cc4568_aae288b7', 'repo_hash': 'dd8d8b52c7702852e93498cf7551667f05cc4568_aae288b7', 'commit_hash': 'dd8d8b52c7702852e93498cf7551667f05cc4568'}) | 15:58 |
| marios|ruck | panda: do you want tostop it and increase verbosity? | 15:58 |
| marios|ruck | panda: master & train should be promoting | 15:59 |
| marios|ruck | oh actually train not yet sec need to check the testproject | 15:59 |
| marios|ruck | panda: master train not yet 17:59 < marios|ruck> panda: master & train should be promoting | 16:00 |
| *** bogdando has quit IRC | 16:01 | |
| marios|ruck | panda: there 17:09 < panda> marios|ruck: I think we need to temprarily increate the verbosity of the ansible playbook | 16:01 |
| weshay|ruck | rlandy, updated | 16:03 |
| rlandy | weshay|ruck: k - let's chat | 16:03 |
| weshay|ruck | rlandy, https://meet.google.com/jih-hvwy-yhw?authuser=1 | 16:03 |
| *** dsneddon has joined #oooq | 16:13 | |
| Tengu | quick question folks: will the tripleo-ci repository be ported to python3? if not, is there a proper way to build packages from git sources? | 16:21 |
| panda | Tengu: I'll give you the wrong answer so somebody here will have to correct me. | 16:28 |
| panda | Tengu: 1) I'ts not in the plans I know of, 2) no | 16:28 |
| Tengu | panda: ;_; | 16:31 |
| Tengu | panda: care to point me the code in the CI where it builds the RPMs? | 16:31 |
| Tengu | because that's supported for centos-8 apparently.... | 16:31 |
| weshay|ruck | Tengu, it's not tripleo-ci | 16:33 |
| weshay|ruck | that builds packages | 16:33 |
| weshay|ruck | for depends-on | 16:33 |
| Tengu | weshay|ruck: any link? the delorean thing was really cool, in tripleo-ci, but apparently it doesn't deploy on centos-8 ^^' | 16:34 |
| weshay|ruck | Tengu, this is what builds the packages.. and consider it deprecated https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/build-test-packages | 16:35 |
| Tengu | weshay|ruck: erf.. soooo what's the new way? :) | 16:35 |
| weshay|ruck | Tengu, what are you looking to do? | 16:35 |
| *** ccamacho has quit IRC | 16:36 | |
| Tengu | weshay|ruck: in my lab, I'm used to get an RPM out of git sources for that or this project (being tripleo-heat-templates, tripleo-ansible, anything) in order to test patches on my env | 16:36 |
| weshay|ruck | k | 16:36 |
| Tengu | weshay|ruck: up to now, I was using the tripleo-ci content, specifically a script named "tripleo.sh" | 16:37 |
| Tengu | and since it doesn't install (missing deps and things) on centos-8 with python3, I'm wondering how CI is doing things now :9 | 16:37 |
| weshay|ruck | Tengu, that hasn't been used in ci for a very long time | 16:39 |
| weshay|ruck | Tengu, no .. there are NO plans to support that bit | 16:39 |
| Tengu | I know that :). still, CI is building things, else it wouldn't be able to test anything right? | 16:39 |
| weshay|ruck | Tengu, the planned way things will get rebuilt would be to integrate the packaging teams roles that build packages into tripleo ci https://pagure.io/zuul-distro-jobs/blob/master/f/roles | 16:39 |
| Tengu | so if you're saying me the oooq-extras thing is being deprecated as well, .. | 16:40 |
| weshay|ruck | Tengu, yes.. atm.. build-test-package role is used but deprecated in favor of https://pagure.io/zuul-distro-jobs/blob/master/f/roles | 16:40 |
| weshay|ruck | chandankumar, ^ | 16:40 |
| weshay|ruck | Tengu, yes | 16:40 |
| Tengu | ah. so that pagure thing. thanks! | 16:40 |
| chandankumar | weshay|ruck, will be deprecated | 16:40 |
| Tengu | I'll check how to integrate those roles. | 16:40 |
| marios|ruck | panda: weshay|ruck: master going again currently 2020-02-24 15:08:05,822 9709 INFO promoter Running: env ANSIBLE_LOG_PATH=/home/centos/promoter_logs/container-push/20200224-150805.log RELEASE=master COMMIT_HASH=da4c8b0acc02c322be90322cef0517f785b14b27 DISTRO_HASH=b5a8f808926e2fb66232f1c5e677c634a807d2dc FULL_HASH=da4c8b0acc02c322be90322cef0517f785b14b27_b5a8f808 PROMOTE_NAME=current-tripleo | 16:41 |
| marios|ruck | SCRIPT_ROOT=/home/centos/ci-config-newcode/ DISTRO_NAME=centos DISTRO_VERSION=7 ansible-playbook -e manifest_push=true -e target_registries_push=true /home/centos/ci-config-newcode/ci-scripts/container-push/container-push.yml | 16:41 |
| marios|ruck | panda: weshay|ruck: if needed you can remote: https://review.rdoproject.org/r/25314 Set manifest_push: false for master promotion debug | 16:41 |
| * marios|ruck shutdown seqence | 16:47 | |
| *** tesseract has quit IRC | 16:47 | |
| weshay|ruck | rlandy, results coming in | 16:48 |
| weshay|ruck | soon | 16:48 |
| rlandy | yep | 16:48 |
| *** marios|ruck is now known as marios|out | 16:50 | |
| rlandy | https://logserver.rdoproject.org/95/24995/18/check/periodic-tripleo-ci-centos-8-standalone-master/570393e/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz | 16:51 |
| rlandy | no dice | 16:51 |
| rlandy | https://logserver.rdoproject.org/95/24995/18/check/periodic-tripleo-ci-centos-8-standalone-master/570393e/logs/undercloud/etc/yum.repos.d/delorean.repo.txt.gz worked | 16:52 |
| weshay|ruck | eek.. give me two seconds | 16:52 |
| weshay|ruck | rlandy, | 16:52 |
| rlandy | priority=100 | 16:52 |
| chandankumar | sorry I am back | 16:53 |
| chandankumar | fixed the podman issue https://github.com/containers/libpod/pull/5312 | 16:53 |
| rlandy | udo sed -i -e "s|priority=1|priority=10|g" /etc/yum.repos.d/delorean-tripleo-current.repo | 16:53 |
| weshay|ruck | chat? | 16:54 |
| rlandy | chandankumar: see weshay|ruck's changes on where the packages are taken from ... | 16:54 |
| rlandy | https://review.opendev.org/#/c/709560/4/config/release/tripleo-ci/CentOS-8/promotion-testing-hash-master.yml | 16:54 |
| rlandy | don;t need that sed | 16:54 |
| rlandy | yeah | 16:54 |
| weshay|ruck | https://meet.google.com/mdi-dboj-umz?authuser=1 | 16:54 |
| rlandy | chandankumar: ^^ | 16:55 |
| *** marios|out has quit IRC | 17:00 | |
| weshay|ruck | chandankumar, rlandy https://review.opendev.org/#/c/709254/2/tripleo_repos/main.py | 17:08 |
| chandankumar | weshay|ruck, sshnaidm podman fix https://github.com/containers/libpod/pull/5312 | 17:22 |
| sshnaidm | chandankumar, why not to check /etc/os-release ? | 17:24 |
| chandankumar | sshnaidm, we can check that | 17:25 |
| sshnaidm | chandankumar, maybe it's easier then multiple greps | 17:25 |
| chandankumar | sshnaidm, may be switch in another PR, with all rhel and centos hacks | 17:26 |
| sshnaidm | chandankumar, ack | 17:26 |
| chandankumar | let me add f31 job for podman | 17:27 |
| chandankumar | weshay|ruck, rlandy https://logserver.rdoproject.org/95/24995/18/check/periodic-tripleo-ci-centos-8-standalone-master/4db3956/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz | 17:31 |
| chandankumar | It is coming due to priority I think | 17:31 |
| chandankumar | sshnaidm, zbr https://review.rdoproject.org/r/25316 | 17:32 |
| rlandy | looking | 17:32 |
| rlandy | https://logserver.rdoproject.org/95/24995/18/check/periodic-tripleo-ci-centos-8-standalone-master/4db3956/logs/undercloud/etc/yum.repos.d/delorean.repo.txt.gz | 17:33 |
| rlandy | 204 | 17:33 |
| rlandy | 20 | 17:33 |
| rlandy | https://logserver.rdoproject.org/95/24995/18/check/periodic-tripleo-ci-centos-8-standalone-master/4db3956/logs/undercloud/etc/yum.repos.d/delorean-tripleo-current.repo.txt.gz | 17:33 |
| rlandy | include packages ia still off | 17:34 |
| rlandy | package ansible-pacemaker-1.0.4-0.20200116145501.fcdc638.el8.noarch is excluded | 17:35 |
| rlandy | what provides ansible-pacemaker | 17:36 |
| weshay|ruck | amoralej, ^ | 17:39 |
| *** jmasud has quit IRC | 17:45 | |
| amoralej | that should come from delorean | 17:46 |
| amoralej | tripleo-current | 17:47 |
| amoralej | oh | 17:47 |
| amoralej | maybe old tripleo-current was pruned for some reason | 17:47 |
| *** jmasud has joined #oooq | 17:48 | |
| chandankumar | rlandy, weshay|ruck I will be not around on friday, need to attendee local conf in Pune | 17:50 |
| rlandy | ok | 17:52 |
| weshay|ruck | k | 17:52 |
| weshay|ruck | rlandy, did that get through? | 17:59 |
| weshay|ruck | ah.. not yet | 17:59 |
| rlandy | still running | 18:00 |
| rlandy | failed | 18:01 |
| rlandy | https://logserver.rdoproject.org/95/24995/18/check/periodic-tripleo-ci-centos-8-standalone-master/6df2fe6/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz | 18:04 |
| rlandy | weshay|ruck: ^^ | 18:04 |
| rlandy | openvswitch | 18:04 |
| weshay|ruck | rlandy, fak me | 18:04 |
| rlandy | see #rdo | 18:05 |
| weshay|ruck | zbr, https://review.opendev.org/#/c/708402/ | 18:06 |
| weshay|ruck | zbr, please review | 18:06 |
| weshay|ruck | rlandy, chandankumar ok.. so we're blocked until that merges | 18:07 |
| rlandy | why? | 18:08 |
| rlandy | depends-on | 18:08 |
| weshay|ruck | rlandy, try it | 18:08 |
| rlandy | already in progress | 18:08 |
| chandankumar | weshay|ruck, rlandy it's fun fun time, so many depends on | 18:08 |
| rlandy | real party here | 18:09 |
| weshay|ruck | rlandy, you updating the testproject? | 18:09 |
| chandankumar | Open shamphane bottle sponsered by weshay|ruck | 18:09 |
| rlandy | we will have to raise all the depends-on patches at the tripleo meeting | 18:09 |
| weshay|ruck | lolz | 18:09 |
| rlandy | weshay|ruck: sone | 18:09 |
| rlandy | done | 18:09 |
| rlandy | rekicked | 18:09 |
| rlandy | weshay|ruck: ^^ can we raise all the depends-on patches at the tripleo meeting - esp those for chandankumar's build images | 18:10 |
| chandankumar | rlandy, +1 | 18:10 |
| chandankumar | weshay|ruck, rlandy what about droping an email to list once we have working stuff? with depends on | 18:12 |
| rlandy | chandankumar: ack | 18:12 |
| rlandy | compile a list f all the patches we need in for centos 8 promotions | 18:12 |
| weshay|ruck | mjturek, hey.. being told cico has centos-8 now | 18:14 |
| weshay|ruck | mjturek, baha https://trello.com/c/fv3u22df/709-centos8-move-to-centos8 | 18:15 |
| weshay|ruck | CentOS | 18:15 |
| weshay|ruck | 8 | 18:15 |
| weshay|ruck | x86_64 | 18:15 |
| weshay|ruck | None | 18:15 |
| weshay|ruck | 15 | 18:15 |
| weshay|ruck | rlandy, we need to install mock directly from epel | 18:21 |
| weshay|ruck | we've done this in the past | 18:21 |
| weshay|ruck | in previous release files | 18:22 |
| weshay|ruck | why mock is not core.. I have no idea | 18:22 |
| rlandy | yeah - getting | 18:23 |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/RedHat-8/master.yml#L105 | 18:24 |
| rlandy | for rhel 8 | 18:24 |
| chandankumar | rlandy, install it via epel | 18:25 |
| chandankumar | then remove it | 18:25 |
| chandankumar | see ya tomorrow | 18:38 |
| *** chandankumar is now known as raukadah | 18:38 | |
| rlandy | weshay|ruck: http://pastebin.test.redhat.com/838968 | 18:40 |
| rlandy | ^^ enough? | 18:40 |
| rlandy | or do we need all of https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/DLRN/setup_mock.yaml | 18:41 |
| weshay|ruck | rlandy, epel release would have to be in the base repos | 18:42 |
| weshay|ruck | rlandy, I don't think either will work | 18:43 |
| weshay|ruck | as writte | 18:43 |
| rlandy | weshay|ruck: let's do this on the tmate | 18:43 |
| weshay|ruck | as written | 18:43 |
| weshay|ruck | k | 18:44 |
| weshay|ruck | https://meet.google.com/vjq-vcnh-qom?authuser=1 | 18:44 |
| weshay|ruck | rlandy, | 18:44 |
| weshay|ruck | rlandy, we could do it in pre... | 18:45 |
| weshay|ruck | before we sudo rm -rf /etc/yum.repos.d/*.rpmsave | 18:45 |
| weshay|ruck | sudo yum-config-manager --disable "*" | 18:45 |
| *** dpawlik has quit IRC | 18:49 | |
| *** amoralej is now known as amoralej|off | 19:00 | |
| weshay|ruck | rlandy, that may have worked | 19:05 |
| rlandy | weshay|ruck: *** looking in disbelief *** | 19:06 |
| rlandy | holy cow!!! | 19:07 |
| raukadah | rlandy, weshay|ruck hope standalone successed tonight | 19:08 |
| rlandy | here's hoping | 19:08 |
| weshay|ruck | rlandy, so.. forward looking.. ensure build_id is used in the container_prep | 19:12 |
| weshay|ruck | rlandy, can you poke at that? | 19:12 |
| weshay|ruck | for next $oops | 19:12 |
| rlandy | weshay|ruck: I can see so far it picked up the right hash | 19:14 |
| rlandy | "container_build_id": "6807e14fa126605830631804b3db2345094bd1af_2689bb69" | 19:14 |
| rlandy | ^^ wrong though | 19:14 |
| rlandy | 2020-02-24 18:58:41.155498 | TASK [get_hash : read md5] | 19:15 |
| rlandy | 2020-02-24 18:58:41.632599 | primary | 0481ffa881dbc1aa61d9810533d85229 | 19:15 |
| rlandy | ^^ right | 19:15 |
| rlandy | checking container_build_id | 19:15 |
| weshay|ruck | aye.. that's the important one :) | 19:16 |
| weshay|ruck | 2020-02-24 19:06:13.717260 | primary | TASK [standalone : update container tag if needed in container-prepare-parameters.yaml] *** | 19:17 |
| weshay|ruck | 2020-02-24 19:06:13.729832 | primary | Monday 24 February 2020 19:06:13 +0000 (0:00:00.584) 0:04:17.200 ******* | 19:17 |
| weshay|ruck | 2020-02-24 19:06:14.131788 | primary | changed: [undercloud] | 19:17 |
| rlandy | docker_image_tag | 19:20 |
| rlandy | docker_image_tag: *promotion-testing-tag | 19:21 |
| weshay|ruck | rlandy, https://logserver.rdoproject.org/95/24995/19/check/periodic-tripleo-ci-centos-8-standalone-master/a6e0143/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz | 19:22 |
| rlandy | weshay|ruck: ack - it comes from what I pasted above | 19:23 |
| rlandy | https://logserver.rdoproject.org/95/24995/19/check/periodic-tripleo-ci-centos-8-standalone-master/a6e0143/logs/undercloud/var/log/extra/dump_variables_vars.json.txt.gz | 19:23 |
| rlandy | weshay|ruck: ^^ search for dlrn_hash | 19:23 |
| weshay|ruck | rlandy, chat | 19:23 |
| rlandy | k | 19:24 |
| weshay|ruck | https://meet.google.com/npw-rjis-ngx?authuser=1 | 19:24 |
| weshay|ruck | rlandy, https://review.opendev.org/709597 | 19:34 |
| *** dpawlik has joined #oooq | 19:45 | |
| rlandy | "container_build_id": "tripleo-ci-testing" | 19:49 |
| rlandy | weshay|ruck: ^^ | 19:49 |
| rlandy | full_hash must not be defined at that point | 19:50 |
| rlandy | getting line to calculate that directly | 19:50 |
| rlandy | weshay|ruck: we need to fix this https://github.com/openstack/tripleo-quickstart/blob/f3cfc172c069ce2b0e68a6ada0b0becfe4ac344f/roles/repo-setup/tasks/get-dlrn-hash.yml#L15 | 19:59 |
| * weshay|ruck looks | 19:59 | |
| weshay|ruck | rlandy, ya.. agreed we need a fix | 20:01 |
| weshay|ruck | rlandy, tmate? | 20:01 |
| rlandy | weshay|ruck: ack | 20:01 |
| rlandy | there's what to do here | 20:01 |
| rlandy | and chat :) | 20:01 |
| weshay|ruck | https://meet.google.com/mmr-wrax-oou?authuser=1 | 20:02 |
| *** jmasud has quit IRC | 20:12 | |
| *** dpawlik has quit IRC | 20:14 | |
| *** jmasud has joined #oooq | 20:14 | |
| weshay|ruck | rlandy, http://10.0.76.142/rhel8-osp17/component/ | 20:22 |
| *** jfrancoa has quit IRC | 20:31 | |
| *** jfrancoa has joined #oooq | 20:47 | |
| *** TrevorV has quit IRC | 21:37 | |
| *** jfrancoa has quit IRC | 21:40 | |
| *** Goneri has quit IRC | 21:40 | |
| weshay|ruck | rlandy, what's your poison? | 22:00 |
| weshay|ruck | 2020-02-24 21:28:56.498317 | primary | TASK [standalone : Deploy the standalone] ************************************** | 22:00 |
| weshay|ruck | 2020-02-24 21:28:56.519688 | primary | Monday 24 February 2020 21:28:56 +0000 (0:00:02.313) 0:07:23.055 ******* | 22:00 |
| weshay|ruck | 2020-02-24 21:57:57.320217 | primary | changed: [undercloud] | 22:00 |
| rlandy | weshay|ruck: what are you offering??? | 22:01 |
| rlandy | os_tempest : Execute tempest tests - wow - long time in waiting to see that | 22:02 |
| *** beagles is now known as beagles_rpm_mode | 22:04 | |
| weshay|ruck | lolz.. seriously.. I may break out a bisil visky | 22:04 |
| rlandy | good - have a double for me | 22:05 |
| weshay|ruck | ah.. ur no fun | 22:05 |
| rlandy | I actually have some vodka in my house my friend gave me last purim ... I might actually drink it | 22:05 |
| weshay|ruck | really.. I think we should | 22:05 |
| rlandy | totally | 22:06 |
| weshay|ruck | if tempest passes.. 100% in | 22:06 |
| * rlandy dusts off shot glasses | 22:06 | |
| rlandy | hmmm .. wondering what guest image tempest is using | 22:08 |
| *** jbadiapa has quit IRC | 22:11 | |
| weshay|ruck | rlandy, 2020-02-24 22:21:33.713443 | primary | undercloud : ok=140 changed=69 unreachable=0 failed=0 skipped=266 rescued=1 ignored=0 | 22:21 |
| weshay|ruck | 2020-02-24 22:21:46.593142 | primary | "success": true, | 22:22 |
| rlandy | I know :) | 22:22 |
| weshay|ruck | rlandy, should we resubmit now? | 22:22 |
| weshay|ruck | rlandy, we had a linter fail.. that I think we fixed | 22:22 |
| rlandy | let's get those logs | 22:22 |
| rlandy | one more minute | 22:22 |
| rlandy | we need proof if we get drunk | 22:22 |
| weshay|ruck | k | 22:22 |
| weshay|ruck | TOTALLY | 22:23 |
| rlandy | weshay|ruck: what do you think about promoting to current-tripleo? | 22:23 |
| rlandy | so we can run component and check? | 22:23 |
| rlandy | after you merge | 22:24 |
| rlandy | periodic-tripleo-ci-centos-8-standalone-masterSUCCESS in 1h 19m 15s | 22:30 |
| weshay|ruck | looks good | 22:30 |
| rlandy | weshay|ruck: ^^ we got it in writing | 22:30 |
| weshay|ruck | rlandy, let's drink... I'm going to send out a birth announcement | 22:31 |
| rlandy | let's get the patch in | 22:31 |
| weshay|ruck | ok | 22:31 |
| weshay|ruck | you on the tmate | 22:31 |
| rlandy | yay!!! | 22:31 |
| rlandy | yes | 22:31 |
| weshay|ruck | 2020-02-24 21:15:47.445562 | ubuntu-bionic | config/release/tripleo-ci/CentOS-8/promotion-testing-hash-master.yml:70:1: [warning] comment not indented like content (comments-indentation) | 22:32 |
| weshay|ruck | 2020-02-24 21:15:47.445580 | ubuntu-bionic | config/release/tripleo-ci/CentOS-8/promotion-testing-hash-master.yml:99:4: [warning] comment not indented like content (comments-indentation) | 22:32 |
| weshay|ruck | 2020-02-24 21:15:47.445594 | ubuntu-bionic | config/release/tripleo-ci/CentOS-8/promotion-testing-hash-master.yml:136:3: [warning] comment not indented like content (comments-indentation) | 22:32 |
| weshay|ruck | 2020-02-24 21:15:47.445607 | ubuntu-bionic | config/release/tripleo-ci/CentOS-8/promotion-testing-hash-master.yml:143:3: [warning] comment not indented like content (comments-indentation) | 22:32 |
| weshay|ruck | rlandy, ^ | 22:32 |
| weshay|ruck | I think those are fixed | 22:32 |
| rlandy | ok - cool | 22:32 |
| *** Goneri has joined #oooq | 22:32 | |
| rlandy | weshay|ruck: after drink ... let's chat about current-tripleo | 22:33 |
| weshay|ruck | k.. back in a sec | 22:33 |
| weshay|ruck | rlandy, ok | 22:35 |
| weshay|ruck | https://meet.google.com/nna-ddai-jdy?authuser=1 | 22:35 |
| rlandy | k | 22:36 |
| *** Goneri has quit IRC | 22:40 | |
| *** tosky has quit IRC | 23:43 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
