| *** irclogbot_1 has joined #oooq | 00:22 | |
| *** irclogbot_1 has quit IRC | 00:40 | |
| *** ysandeep has joined #oooq | 01:10 | |
| *** irclogbot_1 has joined #oooq | 01:26 | |
| *** irclogbot_1 has quit IRC | 01:48 | |
| *** irclogbot_3 has joined #oooq | 02:46 | |
| *** ykarel|away has joined #oooq | 03:03 | |
| *** brault has quit IRC | 03:16 | |
| *** brault has joined #oooq | 03:22 | |
| *** epoojad1 has joined #oooq | 04:09 | |
| *** ykarel|away has quit IRC | 04:12 | |
| *** udesale has joined #oooq | 04:25 | |
| *** ykarel|away has joined #oooq | 04:27 | |
| *** bhagyashris has joined #oooq | 04:29 | |
| *** skramaja has joined #oooq | 04:33 | |
| *** ykarel|away is now known as ykarel | 04:39 | |
| *** surpatil has joined #oooq | 05:15 | |
| *** epoojad1 has quit IRC | 05:20 | |
| *** soniya29 has joined #oooq | 05:33 | |
| *** raukadah is now known as chkumar|ruck | 06:04 | |
| *** dtrainor has quit IRC | 06:24 | |
| *** marios|rover has joined #oooq | 06:28 | |
| chkumar|ruck | marios|rover: Good morning | 06:46 |
|---|---|---|
| chkumar|ruck | marios|rover: the jobs which hit time out in master periodic pipeline is hitting this bug https://bugs.launchpad.net/tripleo/+bug/1855655 | 06:46 |
| openstack | Launchpad bug 1855655 in tripleo "FS01 master overcloud image prepare time out due to 'module' object has no attribute 'abc' in dhcp agent" [Critical,Confirmed] | 06:46 |
| chkumar|ruck | marios|rover: fix is in progress https://review.rdoproject.org/r/#/c/24022/ | 06:47 |
| chkumar|ruck | thanks! | 06:47 |
| *** dtrainor has joined #oooq | 06:47 | |
| *** akahat has joined #oooq | 06:50 | |
| marios|rover | o/ chkumar|ruck | 06:52 |
| marios|rover | chkumar|ruck: sure gimme few thanks for info | 06:53 |
| marios|rover | chkumar|ruck: nice looks like we got train back up and running | 07:02 |
| marios|rover | chkumar|ruck: promotions fri/sat/sun \o/ | 07:02 |
| chkumar|ruck | yup | 07:03 |
| marios|rover | chkumar|ruck: we were chasing it friday cos of rdo issues RETRY_LIMIT etc | 07:03 |
| *** apetrich has joined #oooq | 07:03 | |
| chkumar|ruck | marios|rover: thank you for fixing that, thanks to rlandy and sshnaidm|off alsp :-) | 07:03 |
| marios|rover | chkumar|ruck: (is on the etherpad) | 07:03 |
| marios|rover | chkumar|ruck: didn't fix something rlandy reverted https://review.rdoproject.org/r/#/c/23994/ | 07:04 |
| chkumar|ruck | ah, looking | 07:04 |
| marios|rover | chkumar|ruck: we even hit it in the testproject looks like https://review.rdoproject.org/r/#/c/23995/ | 07:05 |
| marios|rover | chkumar|ruck: problem was growing number of undeleted stack | 07:05 |
| chkumar|ruck | marios|rover: Does the post cleanup script is not deleting stack after revert the change https://review.rdoproject.org/r/#/c/23994/ ? | 07:08 |
| marios|rover | chkumar|ruck: i think its fine now the revert was th fix | 07:08 |
| chkumar|ruck | ok, cool :-) | 07:08 |
| *** apetrich has quit IRC | 07:08 | |
| marios|rover | http://dashboard-ci.tripleo.org/d/YRJtmtNWk/cockpit?orgId=1&fullscreen&panelId=231 | 07:09 |
| marios|rover | chkumar|ruck: ^^ looks calm | 07:09 |
| chkumar|ruck | yes | 07:09 |
| marios|rover | chkumar|ruck: cool your kolla patch merged | 07:22 |
| chkumar|ruck | yes | 07:22 |
| chkumar|ruck | marios|rover: I have rerunned fs01 train anf centos7 full tempest master job in test project | 07:23 |
| marios|rover | chkumar|ruck: ack | 07:25 |
| marios|rover | chkumar|ruck: we have a bug for that failed to glob pattern thing | 07:30 |
| marios|rover | chkumar|ruck: finding | 07:30 |
| chkumar|ruck | ok | 07:30 |
| marios|rover | https://bugs.launchpad.net/tripleo/+bug/1853028 | 07:31 |
| openstack | Launchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged] | 07:31 |
| marios|rover | chkumar|ruck: ^ | 07:31 |
| marios|rover | chkumar|ruck: its not consistent so testproject might pass | 07:31 |
| chkumar|ruck | marios|rover: But it is seen in fs01 train | 07:32 |
| marios|rover | chkumar|ruck: ack add info into bug then | 07:32 |
| chkumar|ruck | marios|rover: done | 07:34 |
| chkumar|ruck | marios|rover: http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/f16cc6f/build.log | 07:34 |
| chkumar|ruck | marios|rover: I am not sure what is the error in this one | 07:34 |
| chkumar|ruck | marios|rover: I am not sure ERROR:root:overcloud-base has no valid mapping for package openssl-perl is related or other message to lvm2 | 07:34 |
| marios|rover | chkumar|ruck: ack thanks i think there was some fix posted to ensure network was started check cix board there is more info there | 07:34 |
| marios|rover | chkumar|ruck: no i think its red herring unrelated | 07:35 |
| *** surpatil is now known as surpatil|lunch | 07:35 | |
| *** ykarel is now known as ykarel|lunch | 07:38 | |
| *** fmount has joined #oooq | 08:00 | |
| *** ysandeep has quit IRC | 08:02 | |
| *** ysandeep has joined #oooq | 08:08 | |
| *** sshnaidm|off is now known as sshnaidm | 08:11 | |
| *** ykarel|lunch is now known as ykarel | 08:11 | |
| *** jtomasek has joined #oooq | 08:15 | |
| *** d0ugal has joined #oooq | 08:19 | |
| marios|rover | chkumar|ruck: can you check https://bugs.launchpad.net/tripleo/+bug/1853978/comments/16 i vaguely remember you pinging me about that. do we need to file a new bug for it? fs1 is still borked and blocks rhel8 | 08:19 |
| openstack | Launchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Triaged] | 08:19 |
| *** tosky has joined #oooq | 08:25 | |
| *** tesseract has joined #oooq | 08:30 | |
| *** surpatil|lunch is now known as surpatil | 08:31 | |
| *** tesseract has quit IRC | 08:31 | |
| *** tesseract has joined #oooq | 08:31 | |
| marios|rover | chkumar|ruck: ah maybe its that one then https://review.rdoproject.org/r/#/c/24022/ https://bugs.launchpad.net/tripleo/+bug/1853978/comments/16 | 08:37 |
| openstack | Launchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Triaged] | 08:37 |
| marios|rover | chkumar|ruck: no sorry wrong bug | 08:37 |
| marios|rover | chkumar|ruck: its not that | 08:37 |
| marios|rover | chkumar|ruck: yeah added comment/16 to etherpad | 08:38 |
| marios|rover | chkumar|ruck: so we likely need new on there | 08:38 |
| chkumar|ruck | marios|rover: I have re-runned the fs01 train rhel8 job, it is still running | 08:39 |
| marios|rover | chkumar|ruck: ack | 08:40 |
| chkumar|ruck | marios|rover: will update the above bug once the run finishes | 08:40 |
| marios|rover | chkumar|ruck: sounds good | 08:41 |
| marios|rover | chkumar|ruck: you running fs1 master somwhere? me scrolls up | 08:44 |
| chkumar|ruck | marios|rover: fs01 rhel8 train https://review.rdoproject.org/r/24027 | 08:45 |
| chkumar|ruck | master full tempest standlone https://review.rdoproject.org/r/#/c/24026/ | 08:45 |
| chkumar|ruck | will kick out all timeout job once https://review.rdoproject.org/r/#/c/24025/ gets merged | 08:45 |
| marios|rover | chkumar|ruck: no master fs1 like for https://bugs.launchpad.net/tripleo/+bug/1855655/comments/4 | 08:46 |
| openstack | Launchpad bug 1855655 in tripleo "FS01 master overcloud image prepare time out due to 'module' object has no attribute 'abc' in dhcp agent" [Critical,Confirmed] | 08:46 |
| chkumar|ruck | marios|rover: for this one, I have not runned any testproject job, proposing right now | 08:47 |
| marios|rover | chkumar|ruck: i see also standalone 1 and scen7 & tempest job | 08:47 |
| marios|rover | chkumar|ruck: i mean in https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master | 08:47 |
| chkumar|ruck | marios|rover: checking | 08:48 |
| marios|rover | chkumar|ruck: its best to log onto promoter and grab the latest centos master criteria there to know which to include | 08:48 |
| *** saneax has joined #oooq | 08:51 | |
| chkumar|ruck | yes sure | 08:51 |
| chkumar|ruck | marios|rover: I am taking a look at scenario7 centos standalone failure | 08:53 |
| marios|rover | chkumar|ruck: ack | 08:56 |
| *** udesale has quit IRC | 09:01 | |
| *** ykarel_ has joined #oooq | 09:03 | |
| *** ykarel has quit IRC | 09:06 | |
| *** yolanda has joined #oooq | 09:09 | |
| *** ysandeep has quit IRC | 09:24 | |
| *** ysandeep_ has joined #oooq | 09:24 | |
| *** ysandeep_ has quit IRC | 09:26 | |
| *** kopecmartin|off is now known as kopecmartin | 09:27 | |
| marios|rover | chkumar|ruck: did you post something already? me starting to review pipelines should I check centos master? | 09:35 |
| marios|rover | chkumar|ruck: actually scratch that need to do some reviews first will do that next | 09:35 |
| marios|rover | chkumar|ruck: but if you posted/about to let me know i'll start at train | 09:36 |
| marios|rover | or rhel | 09:36 |
| *** d0ugal has quit IRC | 09:36 | |
| *** derekh has joined #oooq | 09:38 | |
| *** d0ugal has joined #oooq | 09:38 | |
| *** apetrich has joined #oooq | 09:42 | |
| *** holser has joined #oooq | 09:46 | |
| chkumar|ruck | marios|rover: not posted got into meeting | 09:52 |
| *** epoojad1 has joined #oooq | 09:54 | |
| *** ykarel__ has joined #oooq | 09:54 | |
| *** ykarel__ is now known as ykarel | 09:54 | |
| *** ykarel_ has quit IRC | 09:56 | |
| chkumar|ruck | marios|rover: we need to wait till the rdo current becomes consistent, in another 2 hours promotion jobs will run | 09:57 |
| chkumar|ruck | so sending test project review for master does not help | 09:58 |
| chkumar|ruck | marios|rover: currently openstack-periodic-latest-released train promotion pipeline is running | 10:00 |
| chkumar|ruck | https://review.rdoproject.org/zuul/status | 10:00 |
| *** dtantsur|afk is now known as dtantsur | 10:02 | |
| marios|rover | chkumar|ruck: ack | 10:03 |
| *** udesale has joined #oooq | 10:08 | |
| *** ssbarnea has quit IRC | 10:16 | |
| *** epoojad1 has quit IRC | 10:26 | |
| *** saneax has quit IRC | 10:30 | |
| *** saneax has joined #oooq | 10:30 | |
| *** udesale has quit IRC | 10:31 | |
| chkumar|ruck | marios|rover: standalone centos 7 full tempest was transient it passed in test project http://logs.rdoproject.org/26/24026/1/check/periodic-tripleo-ci-centos-7-standalone-full-tempest-master/d1574a5/logs/stestr_results.html | 10:33 |
| marios|rover | chkumar|ruck: cool | 10:39 |
| marios|rover | chkumar|ruck: so did you already post the master testproject or should i go ahead? | 10:43 |
| chkumar|ruck | marios|rover: it will not help currently as rdo current is not yet consistent | 10:50 |
| chkumar|ruck | we need to wait for 1 hr 10 mins more to wait for promotion job to run | 10:50 |
| marios|rover | chkumar|ruck: ah right | 10:50 |
| marios|rover | chkumar|ruck: rechecking https://review.opendev.org/#/c/696871/2 not sure if that undercloud-upgrades thing is a race seen it before | 10:51 |
| chkumar|ruck | marios|rover: ack | 10:51 |
| marios|rover | chkumar|ruck: and same there https://review.opendev.org/#/c/696872/1 | 10:52 |
| *** matbu has quit IRC | 10:53 | |
| *** matbu has joined #oooq | 10:53 | |
| *** jtomasek has quit IRC | 10:54 | |
| *** jtomasek has joined #oooq | 10:57 | |
| *** ykarel is now known as ykarel|afk | 10:58 | |
| zbr | chkumar|ruck: arxcruz: any reasons for not triggering github-check on pull events? see https://review.rdoproject.org/r/#/c/24032/ | 11:00 |
| chkumar|ruck | zbr: commented | 11:05 |
| *** ssbarnea has joined #oooq | 11:38 | |
| marios|rover | panda: can we access logs from queens promoter? | 11:57 |
| marios|rover | panda: still on new promoter or did you restore queens? | 11:57 |
| chkumar|ruck | marios|rover: filed the new bug https://bugs.launchpad.net/tripleo/+bug/1855706 | 11:59 |
| openstack | Launchpad bug 1855706 in tripleo "FS 01 train rhel8 overcloud deploy failed with failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network: No such file or directory" [Critical,Triaged] | 11:59 |
| marios|rover | panda: looks like it still isn't on old promoter... is the new one running in a service or are you running queens manually? | 12:01 |
| marios|rover | panda: also are you using upstream criteria for that? | 12:01 |
| marios|rover | chkumar|ruck: ack | 12:01 |
| marios|rover | chkumar|ruck: are you sure it isn't duplicate for https://bugs.launchpad.net/tripleo/+bug/1853028 | 12:03 |
| openstack | Launchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged] | 12:03 |
| chkumar|ruck | marios|rover: in RHEL-8 fs01 we are consuming the built overcloud image so it is a different issue all together | 12:04 |
| panda | marios|rover: manually, but I don't think I need to control the promotions there, the new promoter succeeded, with the modifications | 12:05 |
| panda | marios|rover: and the criteria were the one in the latest commit, which probaby is not updated. | 12:06 |
| marios|rover | panda: so once your patch merges ? https://review.rdoproject.org/r/#/c/23931/ | 12:06 |
| marios|rover | panda: we can run it as a service there? | 12:06 |
| marios|rover | panda: more like a discussion we should have on the call in 1 hour | 12:06 |
| marios|rover | panda: can you try re-running a promotion later ... i think we are just missing fs20 there (posted https://review.rdoproject.org/r/#/c/24033/ i can ping you later once it reports but we can discuss first on the phone | 12:08 |
| panda | marios|rover: sure | 12:09 |
| chkumar|ruck | marios|rover: Do we want to bug networking people about the same failure? | 12:09 |
| marios|rover | chkumar|ruck: not sure... find the trello there's some fix and discussion in there. not clear if that is networking or df | 12:10 |
| chkumar|ruck | ok | 12:10 |
| *** ykarel|afk is now known as ykarel | 12:11 | |
| arxcruz | all: I'm in the new office, and it's a little bit messed here still | 12:23 |
| rfolco | same @home here | 12:26 |
| *** udesale has joined #oooq | 12:33 | |
| chkumar|ruck | marios|rover: please have a look at this card https://trello.com/c/WTFR2Z8E/1235-cixlp1853652tripleociproa-openstack-overcloud-node-provide-all-manageable-timing-out-and-failing-in-both-centos7-and-rhel8-jobs | 12:36 |
| chkumar|ruck | marios|rover: I have closed this bug https://bugs.launchpad.net/tripleo/+bug/1855706 in favor of https://bugs.launchpad.net/tripleo/+bug/1853028 | 12:37 |
| openstack | Launchpad bug 1853028 in tripleo "duplicate for #1855706 Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged] | 12:37 |
| openstack | Launchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged] | 12:37 |
| marios|rover | chkumar|ruck: ack | 12:38 |
| marios|rover | chkumar|ruck: noted a couple tempest issues queens/rocky fs20 on ether | 12:39 |
| chkumar|ruck | will take a look, thanks! | 12:43 |
| zbr | chkumar|ruck: marios|rover : activate periodic for podman package: https://review.rdoproject.org/r/#/c/24036/ | 12:51 |
| *** rlandy has joined #oooq | 12:59 | |
| rfolco | scrum time zbr sshnaidm panda arxcruz chkumar|ruck | 13:00 |
| rfolco | panda, joining? | 13:01 |
| *** apetrich has quit IRC | 13:21 | |
| chkumar|ruck | panda: can you take a look why build is failing http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/cd32825/build.log ? | 13:22 |
| *** skramaja has quit IRC | 13:26 | |
| *** zbr has quit IRC | 13:27 | |
| chkumar|ruck | marios|rover: sorry, we missed the current promotion run, it is not yet conistent https://trunk.rdoproject.org/centos7-master/status_report.html | 13:28 |
| *** zbr has joined #oooq | 13:32 | |
| *** epoojad1 has joined #oooq | 13:32 | |
| panda | chkumar|ruck: 2019-12-09 12:29:02.033 | ERROR:root:overcloud-base has no valid mapping for package openssl-perl | 13:35 |
| panda | chkumar|ruck: this is what I see for now, investigating | 13:35 |
| rlandy | marios|rover: chkumar|ruck: anything you want done/watched after you log off/my afternoon hours? | 13:35 |
| *** Goneri has joined #oooq | 13:35 | |
| marios|rover | rlandy: ack thanks will ping you later if so ? | 13:35 |
| rlandy | marios|rover: ack | 13:35 |
| *** bhagyashris has quit IRC | 13:36 | |
| chkumar|ruck | panda: we are seeing the same error for lvm2 also | 13:57 |
| panda | chkumar|ruck: yep | 13:58 |
| ykarel | chkumar|ruck, centos7 master repo is consistent now | 14:01 |
| ykarel | u can ask ops to get the current periodic run aborted | 14:01 |
| ykarel | and get a rerun | 14:01 |
| ykarel | to have results sooner | 14:02 |
| chkumar|ruck | ykarel: marios|rover I am heading home, asked fbo to abort that currently | 14:03 |
| chkumar|ruck | see ya | 14:03 |
| ykarel | chkumar|ruck, ack | 14:03 |
| marios|rover | chkumar|ruck: k | 14:05 |
| *** epoojad1 has quit IRC | 14:19 | |
| *** soniya29 has quit IRC | 14:22 | |
| migi | rlandy: hey have 5min over bluejeans ? | 14:22 |
| migi | rlandy: re: extra component job | 14:23 |
| rlandy | migi: ack | 14:23 |
| migi | rlandy: if you have then bluejenas.com/mpryc | 14:24 |
| rfolco | zbr, need help o/ | 14:29 |
| rfolco | zbr, get a min ? | 14:29 |
| zbr | in 5min ok? | 14:29 |
| sshnaidm | chkumar|ruck, hi, can you please take a look at scenario012 in your time? It fails on tempest networks, not sure what is the problem there, maybe you'll know better | 14:29 |
| rfolco | zbr, ok thanks | 14:31 |
| rfolco | doesn't have to be in live chat, pls take a look at this http://paste.openstack.org/show/787315/ | 14:32 |
| rfolco | zbr, ^ | 14:32 |
| chkumar|ruck | sshnaidm: periodic or upstream? | 14:33 |
| zbr | rfolco: what is th eproble,? the imp module use? | 14:33 |
| zbr | was dlrn_api tested with both py2/py3? | 14:34 |
| rfolco | zbr, good point maybe not | 14:34 |
| rfolco | maybe its not ready for consumption on py3 | 14:34 |
| migi | rlandy: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/ | 14:35 |
| rfolco | zbr, I think I'll move forward without the module, I am spinning wheels there | 14:36 |
| *** ykarel is now known as ykarel|afk | 14:36 | |
| zbr | rfolco: that imp issue is very easy to fix, i am looking now for the codebase of the module | 14:36 |
| zbr | rfolco: give me ~1h... | 14:37 |
| chkumar|ruck | arxcruz: please merge this https://review.rdoproject.org/r/#/c/22862/ | 14:37 |
| zbr | already started | 14:37 |
| rfolco | zbr, if you think its something quick to fix, we can try it. | 14:37 |
| rfolco | zbr, thx | 14:38 |
| zbr | rfolco: is this what you are using https://github.com/softwarefactory-project/dlrnapi_client ? | 14:38 |
| rfolco | yes | 14:39 |
| rfolco | got an example from https://github.com/softwarefactory-project/dlrnapi_client/blob/master/dlrnapi_client/ansible/example_playbook.yaml | 14:39 |
| *** TrevorV has joined #oooq | 14:39 | |
| rfolco | zbr, and then switched to py3 on ansible.cfg | 14:40 |
| sshnaidm | chkumar|ruck, check job, it doesn't run on periodic | 14:42 |
| chkumar|ruck | sshnaidm: check checking | 14:42 |
| chkumar|ruck | is it the ironic one? | 14:42 |
| chkumar|ruck | sshnaidm: it is failing from very long time | 14:43 |
| sshnaidm | chkumar|ruck, yeah, and Ironic part should be solved now | 14:43 |
| chkumar|ruck | checking now | 14:44 |
| *** soniya29 has joined #oooq | 14:44 | |
| *** akahat has quit IRC | 14:50 | |
| *** akahat has joined #oooq | 14:51 | |
| *** soniya29 has quit IRC | 14:52 | |
| *** surpatil has quit IRC | 14:54 | |
| chkumar|ruck | marios|rover: we have killed master periodic pipeline | 14:59 |
| rfolco | zbr, didn't work on py3 :( | 15:01 |
| marios|rover | chkumar|ruck: yeah you said before... there is one more run today so lets see | 15:02 |
| marios|rover | in like 3 hours | 15:02 |
| chkumar|ruck | marios|rover: we killed it right now | 15:03 |
| marios|rover | chkumar|ruck: oh you mean 16:03 #oooq: < chkumar|ruck> ykarel: marios|rover I am heading home, asked fbo to abort that currently | 15:03 |
| marios|rover | chkumar|ruck: didn't work 1 hour ago? | 15:03 |
| chkumar|ruck | marios|rover: nope, he killed right now | 15:03 |
| marios|rover | chkumar|ruck: ack ok. same though there is one more in ~ 3 hours | 15:04 |
| chkumar|ruck | marios|rover: let's watch them tomorrow | 15:04 |
| marios|rover | chkumar|ruck: yeah we'll see results of tonights run in our morning | 15:04 |
| chkumar|ruck | :-) | 15:04 |
| chkumar|ruck | arxcruz: still around? | 15:04 |
| chkumar|ruck | arxcruz: I am going to remove smoke tests from herhttps://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml#L23 and running it only in periodic as it takes too much time | 15:05 |
| chkumar|ruck | as we are depreacting os_tempest job | 15:05 |
| chkumar|ruck | rfolco: zbr https://github.com/rdo-infra/weirdo/blob/master/playbooks/dlrn-api-report.yml#L31 I am not sure it will be useful for you related to dlrn api client | 15:08 |
| rfolco | chkumar|ruck, does it work on py3 ? | 15:09 |
| chkumar|ruck | rfolco: need to check with dmsiard | 15:09 |
| rfolco | chkumar|ruck, thats what zbr is testing/fixing | 15:09 |
| rfolco | thx for the pointer chkumar|ruck | 15:10 |
| chkumar|ruck | rfolco: poked dms on #rdo channel | 15:10 |
| rlandy | migi: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml | 15:12 |
| chkumar|ruck | marios|rover: sshnaidm rfolco rlandy zbr see ya tomorrow, will fix stuff tomorrow | 15:18 |
| *** chkumar|ruck is now known as raukadah | 15:19 | |
| marios|rover | have a good one raukadah | 15:19 |
| *** ykarel|afk is now known as ykarel|away | 15:25 | |
| zbr | rfolco: create a card for dlrn_api py3 support and assign it to me, i know what needs to be done. | 15:26 |
| *** ykarel|away has quit IRC | 15:31 | |
| *** ykarel|away has joined #oooq | 15:31 | |
| *** ykarel|away has quit IRC | 15:32 | |
| *** udesale has quit IRC | 15:36 | |
| rfolco | zbr, you rock man | 15:37 |
| panda | anyone care to give a last review and +2 ? https://review.rdoproject.org/r/23931 | 15:38 |
| rfolco | zbr, https://tree.taiga.io/project/tripleo-ci-board/task/1429?kanban-status=1447275 | 15:38 |
| rfolco | zbr, sshnaidm rlandy arxcruz https://review.rdoproject.org/r/23931 | 15:39 |
| rfolco | pls review panda's patch asap | 15:40 |
| rlandy | there's a lot there | 15:43 |
| *** saneax has quit IRC | 15:53 | |
| mjturek | asked on friday but figure I'd ask again since there are more people here | 16:11 |
| mjturek | has anyone seen this error in the container build job before "2019-12-09 12:26:40 | ERROR:kolla.common.utils.base:The command '/bin/sh -c curl -L http://172.17.0.1/delorean.repo -o /etc/yum.repos.d/delorean.repo' returned a non-zero code: 7" | 16:12 |
| mjturek | seeing it here https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1820/logs/logs/build.log | 16:12 |
| mjturek | as far as I can tell the docker bridge is up and pingable, but haven't tried from a container | 16:12 |
| mjturek | also the curl works locally, again didn't try from a container | 16:13 |
| mjturek | baha ^ | 16:14 |
| *** akahat has quit IRC | 16:16 | |
| *** soniya29 has joined #oooq | 16:20 | |
| marios|rover | panda: fyi https://review.rdoproject.org/r/#/c/24033/ green if you have time even tomorrow to run the queens again thanks | 16:26 |
| rfolco | marios|rover, raukadah panda have you seen this error before ? see mjturek message above | 16:28 |
| marios|rover | rfolco: no also replied on friday not seen that one in jobs | 16:29 |
| rfolco | thx marios|rover | 16:29 |
| * marios|rover shutdown sequence | 16:29 | |
| *** d0ugal has quit IRC | 16:29 | |
| mjturek | yep thanks marios|rover and rfolco :) | 16:29 |
| rfolco | mjturek, perhaps your theory about silent error is correct, now need to investigate it | 16:30 |
| *** d0ugal has joined #oooq | 16:31 | |
| raukadah | rfolco: is it possible to hold the node? where failure is coming? | 16:35 |
| raukadah | may be we can jump an debug | 16:36 |
| raukadah | jpena|off will be coming tomorrow, I will ask for help for sure | 16:36 |
| mjturek | raukadah well need to adjust the job to hold on failure but i think e can do that | 16:37 |
| mjturek | you'll need access to cico nodes | 16:37 |
| raukadah | mjturek: I will poke tomorrow a little early tomorrow then | 16:37 |
| mjturek | baha ^ lets do tghe thing we discussed thurs | 16:37 |
| mjturek | adjust the job to hold node on fail | 16:38 |
| *** marios|rover is now known as marios|out | 16:39 | |
| baha | OK! We're going to need to make the job run less frequently in the meantime to not eat all of cico's nodes | 16:39 |
| *** soniya29 has quit IRC | 16:47 | |
| *** marios|out has quit IRC | 16:47 | |
| *** holser has quit IRC | 17:13 | |
| zbr | rfolco: re dlrn, https://softwarefactory-project.io/r/#/c/17161/ and its dependency are ready. | 17:18 |
| rfolco | zbr, wow | 17:18 |
| rfolco | zbr, looking | 17:18 |
| zbr | rfolco: fwi, i does not fix that specific issue, for the moment only increasing the test matrix | 17:19 |
| *** ykarel has joined #oooq | 17:19 | |
| rfolco | zbr, this puts it in py3 game | 17:19 |
| rfolco | ok | 17:19 |
| zbr | mainly i had to add centos-8 testing, we didn't even had the nodeset definition. | 17:19 |
| *** ykarel is now known as ykarel|away | 17:19 | |
| zbr | once i get this merged, i go to next step. which is to replicate that bug, and to fix it. | 17:20 |
| zbr | but w/o job running on py3 i cannot fix it correctly. | 17:20 |
| zbr | it would be more of a guessing game | 17:20 |
| rfolco | zbr, nice, lets get jpena on it | 17:20 |
| zbr | rfolco: where is your call to "dlrn_api" module? | 17:25 |
| rfolco | zbr, I am trying it manually in a local test.... | 17:26 |
| rfolco | zbr, let me get the paste again | 17:26 |
| zbr | i have the paste but is not enough | 17:26 |
| zbr | i need context | 17:26 |
| zbr | the real error is that it fails to find the dlrn_client module, because you did not install it. | 17:26 |
| rfolco | zbr, so I created a virtualenv | 17:27 |
| rfolco | zbr, installed dlrapi_client w/ pip | 17:27 |
| rfolco | zbr, then I created the ansible.cfg on it and the playbook... | 17:27 |
| zbr | it needs to be inside venv used by ansible, ansible task will try to run on remove machine | 17:27 |
| zbr | share the code | 17:27 |
| rfolco | http://paste.openstack.org/show/787315/ | 17:27 |
| zbr | this is not the code, this is the output | 17:28 |
| rfolco | see | 17:28 |
| rfolco | dlrn_client.yaml | 17:28 |
| rfolco | I did a cat with the ansible play there | 17:28 |
| rfolco | :) | 17:28 |
| zbr | i do not see any code insall dlrn_api module | 17:28 |
| rfolco | ah | 17:28 |
| rfolco | so | 17:28 |
| zbr | sorry, dlnr_client module, python module | 17:28 |
| rfolco | I did it manually sorry | 17:28 |
| rfolco | inside the venv | 17:28 |
| rfolco | installed dlrnapi_client | 17:29 |
| zbr | you fixed ansible ability to find its own module, but this ansible module needs the python module inside the same env as the one used by ansible | 17:29 |
| zbr | where? | 17:29 |
| rfolco | line #41 | 17:29 |
| rfolco | parsing your comment, maybe I did something wrong with venv | 17:30 |
| zbr | add this line: command: {{ ansible_python_interpreter }} -c "import dlrn_client" | 17:30 |
| zbr | and you will find the issue | 17:30 |
| rfolco | zbr, as an additional task? | 17:30 |
| zbr | yeah, for debuggin. btw, that pastebin sucks you need to do a lot of horizontal scrolling to see real error | 17:31 |
| zbr | no wrapping | 17:31 |
| zbr | ModuleNotFoundError: No module named 'dlrnapi_client' | 17:31 |
| zbr | if you run ansible with -vvv you will also see the python import paths. | 17:32 |
| zbr | you may be able to address the issue by adding "connection: local" | 17:33 |
| *** dtantsur is now known as dtantsur|afk | 17:33 | |
| zbr | putting localhost does not mean it will connect directly, it will still ssh and not use your venv. | 17:33 |
| zbr | in fact is very easy to test using ad-hoc: just run: ansible -m dlnr_api | 17:34 |
| zbr | probably you will get the same exception | 17:34 |
| rfolco | hmm | 17:34 |
| rfolco | sec | 17:34 |
| rfolco | yes ansible -m dlrn_api localhost retrurned me : No module named 'dlrnapi_client' | 17:36 |
| rfolco | zbr, ^ | 17:36 |
| zbr | exactly what I said | 17:36 |
| zbr | python fails to find the module in its path | 17:36 |
| zbr | ansible library path != python path | 17:37 |
| zbr | but what I succesfully did in the past was to use another ansible task to install that module | 17:38 |
| zbr | because is published on pypa is quite easy | 17:38 |
| rfolco | was hoping this to help | 17:41 |
| rfolco | library = /usr/lib/python3.7/site-packages/dlrnapi_client/ansible:$VIRTUAL_ENV/lib/python3.7/site-packages/dlrnapi_client/ansible | 17:41 |
| rfolco | finding the library | 17:41 |
| zbr | rfolco: see https://github.com/ssbarnea/harem/commit/d76226403c9b555e3c1f877d9dd1f20cf39047e6#diff-7aba2956bf0cba5d967cef55dfc34d73R34-R38 | 17:41 |
| zbr | library is for *ansible* modules, not python modules. | 17:41 |
| rfolco | isn't what we are using here ? | 17:42 |
| zbr | we have both | 17:42 |
| rfolco | so the import inside the module is failing | 17:42 |
| rfolco | is it? | 17:42 |
| zbr | dlnr_api ansible module which needs dnr_client pythob module | 17:42 |
| zbr | yep | 17:42 |
| rfolco | k | 17:42 |
| zbr | it finds the ansible module, runs it and faist to import python module at the begening. | 17:42 |
| rfolco | zbr, so i add a pip module before, and then re-use the same venv | 17:43 |
| rfolco | in the next step | 17:43 |
| zbr | my example may confuse you because the two modules happen to have the same name, but for dlnr is not the same. | 17:43 |
| rfolco | is that what you suggest? | 17:43 |
| zbr | i prefer to use pip from ansible | 17:43 |
| zbr | see if it works | 17:43 |
| rfolco | k | 17:44 |
| zbr | i really do not understand why ansible team refused to make pure python modules, it would so easy to install them. | 17:45 |
| zbr | i asked them and they didn't want it | 17:45 |
| rfolco | zbr, so I added and it still complains. Maybe I need to inform the virtualenv previously used in the pip step ? | 17:46 |
| zbr | let me try to make a test playbook | 17:47 |
| *** dsneddon has joined #oooq | 17:48 | |
| zbr | rfolco: worked for me just fine | 17:55 |
| zbr | https://seashells.io/v/3v5FbYRg | 17:56 |
| zbr | rfolco: but what you get may be different, if ansible is installed as system package you may get a very different experience | 17:57 |
| rfolco | zbr, pls share code not result :) | 17:59 |
| zbr | ansible -m dlrn_api localhost -vv -a "action=repo-status host=foo" | 18:00 |
| *** derekh has quit IRC | 18:01 | |
| zbr | rfolco: another question on which OS are you running and with with version of ansible, i would not be surprised to discover that you have a mix | 18:03 |
| rfolco | zbr, still getting the imp import issue | 18:04 |
| zbr | that is only a warning that you can ignore, scroll for the real error | 18:05 |
| rfolco | no module named 'dlrnapi_client' | 18:05 |
| rfolco | :( | 18:05 |
| rfolco | I am sick of this | 18:05 |
| rfolco | will stick with dlrnapi_client and leave this for future improvement | 18:05 |
| zbr | when it will packed as a collection it should be easier. | 18:06 |
| rfolco | zbr, ok cool, thanks for helping, and lets try to get it working for v2 next sprint | 18:06 |
| zbr | sure, time to go | 18:07 |
| rfolco | zbr, cool thx | 18:15 |
| zbr | rlandy: please put a +2 on https://review.rdoproject.org/r/#/c/24036/ to merge it. | 18:22 |
| rlandy | done | 18:24 |
| *** d0ugal has quit IRC | 18:33 | |
| rlandy | rfolco: any issue with my merging https://review.rdoproject.org/r/#/q/topic:keystone-pipeline+(status:open+OR+status:merged)? | 18:52 |
| rlandy | should not impact anything else | 18:52 |
| rfolco | rlandy, should be fine | 18:53 |
| rlandy | rfolco: can you vote for some legitimacy? | 18:54 |
| *** ykarel|away has quit IRC | 18:54 | |
| rfolco | rlandy, sure | 18:54 |
| rfolco | rlandy, done, +1, one minor question, rebase. Merge. | 18:59 |
| rlandy | rfolco: that was the original plan | 18:59 |
| rlandy | I'm fine to redo it when we redesign in next sprint | 19:00 |
| rlandy | as weshay_ said this is POC | 19:00 |
| rlandy | we will redo it all | 19:00 |
| rfolco | ok, no big deal, just need to think about scaling it | 19:00 |
| rfolco | thx | 19:00 |
| rlandy | rfolco: yep - reasonable question | 19:02 |
| *** dsneddon has quit IRC | 19:16 | |
| *** dsneddon has joined #oooq | 19:19 | |
| *** dsneddon has quit IRC | 19:20 | |
| *** dsneddon has joined #oooq | 19:20 | |
| *** tesseract has quit IRC | 19:37 | |
| mjturek | rfolco: here's a patch that will hold two nodes a day from baha https://review.rdoproject.org/r/#/c/24046/ | 19:58 |
| rfolco | will check it asap, hands tied right now | 20:01 |
| baha | rfolco: Hold off, we found out cico's CLI doesn't actually expose the fail state so we need to write an API call | 20:03 |
| rfolco | baha, mjturek ok I don't know how cico works btw :( | 20:03 |
| rlandy | rfolco: pls check me ... https://review.rdoproject.org/r/24047 Use tripleo-ci-testing for the component pipeline | 20:04 |
| mjturek | rfolco thanks, we're looking into how to handle it with curl :( https://wiki.centos.org/QaWiki/CI/Duffy | 20:04 |
| rlandy | zbr: https://review.rdoproject.org/r/#/c/23924/22/zuul.d/projects.yaml | 20:10 |
| rlandy | pls add files to the podman tests | 20:11 |
| rlandy | they are running on all rdo-jobs changes | 20:11 |
| rlandy | not necessary | 20:11 |
| baha | rfolco: https://review.rdoproject.org/r/#/c/24046/ updated w/ manual API call, should be good | 20:26 |
| mjturek | rfolco basically this will let us take a look at the node after the failure for a couple hours. Should be useful for tomorrow ^ | 21:04 |
| rfolco | baha, mjturek +2, if there is a way to test or show that it works please indicate in the patch. Also, get a revert prepared. | 21:12 |
| rfolco | thx | 21:13 |
| rfolco | otherwise lets just +w and watch | 21:13 |
| mjturek | baha is submitting a revert patch quick | 21:20 |
| mjturek | we don't have any testing to show it works other than a curl to the cico admin server | 21:20 |
| mjturek | a test curl that worked | 21:20 |
| baha | rfolco: Revert's up at https://review.rdoproject.org/r/24048 | 21:25 |
| mjturek | rfolco willing to wf it? | 21:34 |
| rfolco | yes sorry | 21:35 |
| rfolco | was finishing up a patch | 21:35 |
| rfolco | doing now | 21:35 |
| rfolco | mjturek, baha sone | 21:35 |
| rfolco | done | 21:35 |
| mjturek | thanks! | 21:35 |
| rfolco | shutdown sequence in a few | 21:35 |
| mjturek | lolol | 21:35 |
| rfolco | let me know if any others to review | 21:35 |
| rfolco | :) | 21:35 |
| mjturek | will do! | 21:36 |
| baha | thank you rfolco | 21:37 |
| rfolco | yw | 21:42 |
| *** jtomasek has quit IRC | 21:45 | |
| *** Goneri has quit IRC | 21:56 | |
| *** brault has quit IRC | 21:59 | |
| *** dtrainor has quit IRC | 22:06 | |
| *** ysandeep has joined #oooq | 22:22 | |
| *** ysandeep has quit IRC | 22:50 | |
| *** TrevorV has quit IRC | 23:00 | |
| *** rlandy has quit IRC | 23:43 | |
| *** tosky has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!