| *** rlandy|bbl is now known as rlandy | 01:37 | |
| *** rlandy has quit IRC | 01:42 | |
| *** apetrich has quit IRC | 02:16 | |
| *** ykarel has joined #oooq | 03:24 | |
| *** ykarel_ has joined #oooq | 03:31 | |
| *** ykarel has quit IRC | 03:33 | |
| *** skramaja has joined #oooq | 03:49 | |
| *** gkadam has joined #oooq | 03:52 | |
| *** ykarel__ has joined #oooq | 03:58 | |
| *** ykarel_ has quit IRC | 03:58 | |
| *** ykarel__ has quit IRC | 04:22 | |
| *** kopecmartin has joined #oooq | 04:50 | |
| *** ykarel__ has joined #oooq | 04:58 | |
| *** ykarel__ is now known as ykarel | 05:00 | |
| *** jbadiapa has quit IRC | 05:01 | |
| *** ccamacho has joined #oooq | 05:38 | |
| *** udesale has joined #oooq | 05:53 | |
| *** udesale has quit IRC | 06:07 | |
| *** sanjayu_ has joined #oooq | 06:08 | |
| *** udesale has joined #oooq | 06:08 | |
| *** sanjayu__ has joined #oooq | 06:09 | |
| *** sanjayu_ has quit IRC | 06:12 | |
| *** jfrancoa has joined #oooq | 06:21 | |
| *** apetrich has joined #oooq | 07:09 | |
| *** jbadiapa has joined #oooq | 07:14 | |
| *** tesseract-1 has joined #oooq | 07:35 | |
| *** tesseract-1 has quit IRC | 07:36 | |
| *** tosky has joined #oooq | 07:38 | |
| ykarel | ssbarnea, there seems to be an issue with promoter(rocky current-tripleo not promoted to current-tripleo-rdo), there is a traceback: http://38.145.34.55/rocky.log | 08:20 |
|---|---|---|
| *** ykarel is now known as ykarel|lunch | 08:30 | |
| ssbarnea | i am not sure how to deal with this but I will raise bug for it | 08:36 |
| ssbarnea | ykarel|lunch: https://bugs.launchpad.net/tripleo/+bug/1789852 | 08:41 |
| openstack | Launchpad bug 1789852 in tripleo "promoter failure: KeyError: u'current-tripleo-rdo-internal'" [Undecided,New] | 08:41 |
| ssbarnea | it seems that I do not have permissions to edit more bug fieds, maybe someone can add me to tripleo project in LP. | 08:42 |
| *** dtantsur|afk is now known as dtantsur | 08:49 | |
| panda | mmmh | 08:51 |
| panda | interesting | 08:52 |
| *** panda is now known as panda|rover | 08:52 | |
| panda|rover | ssbarnea: are you ruck right now ? | 08:53 |
| ssbarnea | panda|rover: i think so, nobody told me to stop. | 08:54 |
| panda|rover | lol | 08:54 |
| panda|rover | ssbarnea: have you been ruck for how long ? | 08:54 |
| ssbarnea | this is 2nd week. | 08:55 |
| panda|rover | ssbarnea: since the start then ? | 08:55 |
| ssbarnea | panda|rover: yep, since Aug 20 and I was expecting till Sep 7, right? Now I relied mostly on Wes as I do not know lots of things. | 08:58 |
| panda|rover | ssbarnea: I'm your backup, sorry for yesterday, difficult time at home, wanna chat ? | 08:59 |
| *** holser_ has joined #oooq | 09:02 | |
| *** holser_ has quit IRC | 09:02 | |
| *** holser_ has joined #oooq | 09:03 | |
| ssbarnea | panda|rover: https://bluejeans.com/2655417928 | 09:07 |
| *** ykarel|lunch is now known as ykarel | 09:18 | |
| *** udesale has quit IRC | 09:26 | |
| panda|rover | ssbarnea: http://rhos-release.virt.bos.redhat.com:3030/rhosp | 09:37 |
| *** ccamacho has quit IRC | 09:39 | |
| *** ccamacho has joined #oooq | 09:39 | |
| ykarel | ssbarnea, Thanks, good to have a bug for the other issue as well, virtualbmc one as that's also blocking promotion | 09:40 |
| jaosorior | Has anybody seen this error? http://paste.openstack.org/show/729121/ | 10:16 |
| rascasoft | panda|rover, sshnaidm hey folks, can you help me make these https://review.openstack.org/#/c/597432/ https://review.openstack.org/#/c/596789/ move on? It should be a matter of a workflow +1 | 10:22 |
| *** jaosorior has quit IRC | 10:25 | |
| *** jaosorior has joined #oooq | 10:27 | |
| panda|rover | ssbarnea: https://ci.centos.org/job/rdo_trunk-promote-rocky-current-tripleo/ | 10:33 |
| *** udesale has joined #oooq | 10:33 | |
| ykarel | jaosorior, yes we are seeing in rdo phase 1 after virtualbmc 1.4.0, effort for fixing here: https://review.openstack.org/#/c/598095/1 | 10:38 |
| jaosorior | ykarel: thanks | 10:39 |
| ykarel | jaosorior, do u have setup that has the issue, etingof might want to see if his patch is working | 10:40 |
| ykarel | as in ci there is no libvirt job | 10:41 |
| jaosorior | I'm redeploying right now | 10:41 |
| ykarel | kk | 10:42 |
| jaosorior | but yeah, I can check out the patch | 10:42 |
| jaosorior | after this redeploy | 10:42 |
| jaosorior | (and after lunch :D ) | 10:42 |
| ykarel | okk there is a job for ocata, checking logs | 10:42 |
| ykarel | but we wanted master ones | 10:43 |
| ykarel | as virtualbmc is promoted there only | 10:43 |
| *** udesale has quit IRC | 10:44 | |
| *** udesale has joined #oooq | 10:44 | |
| ssbarnea | panda|rover: give this a kick, https://review.openstack.org/#/c/598089/ | 10:57 |
| panda|rover | ssbarnea: approved | 11:00 |
| panda|rover | rascasoft: I'm putting a fix on the promotion and then will look at your patches | 11:00 |
| panda|rover | ssbarnea: https://review.rdoproject.org/r/16042 | 11:10 |
| *** holser_ has quit IRC | 11:14 | |
| *** holser_ has joined #oooq | 11:15 | |
| panda|rover | rascasoft: are you willing to give me some context ? | 11:19 |
| panda|rover | rascasoft: in upgrades images need to be downloaded twice | 11:19 |
| panda|rover | rascasoft: is there another place where we get the second image ? | 11:19 |
| rascasoft | panda|rover, oh I think this answers what me and sshnaidm were discussing about this patch | 11:22 |
| panda|rover | yeah because it needs to download the image for the other release | 11:23 |
| rascasoft | panda|rover, and *just* the overcloud image? Not the ironic-python-agent one? | 11:24 |
| panda|rover | rascasoft: we need to ask upgrades, the patch was added by matby | 11:24 |
| panda|rover | matbu | 11:24 |
| rascasoft | panda|rover, understood | 11:24 |
| panda|rover | I think there may be a better way to do it, but they need to be involved | 11:24 |
| rascasoft | panda|rover, I'll do it | 11:24 |
| panda|rover | rascasoft: thanks | 11:25 |
| rascasoft | panda|rover, by the way, the one I care most is https://review.openstack.org/#/c/597432/ | 11:25 |
| ssbarnea | just discovered how ancient is the gerrit version on rdo... 2.11, fails to even do case-insensitive user lookup... | 11:25 |
| rascasoft | panda|rover, which I just saw it was workflowed. So I'm good thanks! | 11:25 |
| panda|rover | rascasoft: yep, and I think ocata alrady has the new cli anyway | 11:26 |
| *** sshnaidm is now known as sshnaidm|afk | 11:28 | |
| *** dmellado has quit IRC | 11:50 | |
| jaosorior | ykarel: tried out the patch and had the following error: http://paste.openstack.org/show/729129/ | 12:04 |
| *** trown|outtypewww is now known as trown | 12:06 | |
| *** udesale has quit IRC | 12:08 | |
| *** udesale has joined #oooq | 12:11 | |
| *** rlandy has joined #oooq | 12:37 | |
| *** gkadam has quit IRC | 12:38 | |
| rlandy | panda: we are not looking good :( | 12:39 |
| rlandy | couldn't get rdocloud results yesterday | 12:40 |
| rlandy | panda|rover: ^^ | 12:40 |
| *** udesale has quit IRC | 12:40 | |
| panda|rover | rlandy: infra problems ? | 12:44 |
| *** udesale has joined #oooq | 12:53 | |
| rlandy | yep | 12:56 |
| rlandy | we have failures | 12:56 |
| rlandy | but hard to tell what the cause is | 12:56 |
| rlandy | ssbarnea here? | 12:56 |
| ssbarnea | rlandy: yep, here. just raised https://bugs.launchpad.net/tripleo/+bug/1789898 | 12:57 |
| openstack | Launchpad bug 1789898 in tripleo "gate failing due to missing python-tripleoclient" [Undecided,Triaged] - Assigned to Gabriele Cerami (gcerami) | 12:57 |
| rlandy | let's talk at the meeting | 12:57 |
| panda|rover | rlandy: meeting ? | 13:03 |
| *** myoung|pto is now known as moyung | 13:03 | |
| *** moyung is now known as myoung | 13:03 | |
| myoung | o/ | 13:03 |
| *** jbadiapa has quit IRC | 13:03 | |
| *** jbadiapa has joined #oooq | 13:22 | |
| panda|rover | rlandy: rfolco I will put the changes for the vxlan to be backwards compatible, is that ok ? | 13:47 |
| rlandy | panda|rover: fine by me | 13:48 |
| rlandy | I'm starting to log the failures in the card so we can track them | 13:48 |
| ssbarnea | btw, do we have a mongodb deployment up running that I can use for a very small db? if not, where can I deploy an instance? (don't want to use my personal rdo-cloud account for multiple reasons, including stability) | 13:48 |
| rlandy | ssbarnea: do you have a minidell? | 13:49 |
| *** udesale has quit IRC | 13:49 | |
| rlandy | otherwise we can assign you some hardware | 13:49 |
| rlandy | also we have internal cloud space I can lend you for a bit | 13:50 |
| rlandy | we have a test tenant | 13:50 |
| *** udesale has joined #oooq | 13:50 | |
| panda|rover | oh we have to refactor everything | 13:51 |
| panda|rover | oh joy | 13:51 |
| panda|rover | we have logic in the variables currently | 13:51 |
| panda|rover | there's no way i can put a conditional os.stat("/etc/nodepool") in those variables | 13:52 |
| ssbarnea | rlandy: nope. is not that I don't have a place where to install mongodb, i have plenty but I want one that is production rated and that not depends on one individual. | 13:52 |
| ssbarnea | on downstream we had a team cloud account used for services, perfect for this use case, this being different than the tennant accounts used for testing. | 13:53 |
| ssbarnea | in the end, even rdo cloud would be ok, but it must be team tennant. i wipe my own account too often to trust putting a database on it :D | 13:54 |
| rlandy | ssbarnea: we have a team account | 13:54 |
| rlandy | but I'm using it | 13:54 |
| rlandy | depending on how long you need it for | 13:54 |
| rlandy | we can accommodate you | 13:54 |
| ssbarnea | rlandy: likely long term, but 2GB RAM is more than enough. | 13:56 |
| rlandy | ssbarnea: long term - no good there - too much chance it will get wiped | 13:59 |
| rlandy | ssbarnea: when d you need this by? we are early adopters of upshift | 14:00 |
| *** ykarel is now known as ykarel|away | 14:00 | |
| rlandy | best I can offer atm is our stage tenant | 14:00 |
| rlandy | which is also a team tenant | 14:00 |
| rlandy | I am the only one who uses it | 14:00 |
| rlandy | so pretty safe | 14:00 |
| rlandy | if you can hang on, I'll pull you into our upshift adoption work | 14:01 |
| ssbarnea | rlandy: well, it may work. mainly I want to use it to keep a BFA database for Jenkins, which stores one small json object for each build. the DB is for stats and even if goes down it does not impact jenkins (may miss some builds in worst case), still I don't want to hear that VM vanished by accident. Shortly uptime is less important than the historical data recorded there. Yep, i can wait. | 14:04 |
| *** ykarel|away has quit IRC | 14:06 | |
| ssbarnea | rlandy: in fact I can have a different approach as free tier from https://www.mongodb.com/cloud/atlas/pricing would cover for at least 6mo if not even more than one year. | 14:07 |
| rlandy | ssbarnea: k- ping me when this sf fun is over and we will work something out for you | 14:07 |
| ssbarnea | rlandy: sure. | 14:08 |
| ssbarnea | btw, as a team do we use any safe medium for sharing credentials? like ansible-vaulted repo, 1password, lastpass, ... | 14:09 |
| panda|rover | ssbarnea: IRC | 14:09 |
| rlandy | ssbarnea: nah - we just trust each other | 14:09 |
| rlandy | very safe | 14:10 |
| ssbarnea | panda|rover: LOL, "irc - made with security in mind" :D | 14:10 |
| *** sshnaidm|afk has quit IRC | 14:10 | |
| panda|rover | ssbarnea: well, we usually discover security problems only every six months, right before the PTG, so I guess it's OK. | 14:12 |
| panda|rover | myoung: do you know Terry Pratchett and the Unseen University ? | 14:14 |
| rlandy | panda|rover: when you are done rebasing the world, pls vote on https://review.openstack.org/#/c/581488 | 14:15 |
| panda|rover | rlandy: refactoring. I have to change everything. | 14:16 |
| rlandy | happiness is ... | 14:16 |
| *** sshnaidm|afk has joined #oooq | 14:19 | |
| *** sshnaidm|afk is now known as sshnaidm | 14:20 | |
| myoung | panda|rover: I know of the author, but have not read him yet | 14:23 |
| *** ykarel has joined #oooq | 14:35 | |
| rlandy | rascasoft: hi - do you ever see problems installing the undercloud on rhos-13? | 14:45 |
| rascasoft | rlandy, I'd say no. Last time I tried everything went fine. I just have problems in rdocloud because of the well known issue of the getcert | 14:46 |
| rascasoft | rlandy, last time I tried -> on baremetal | 14:47 |
| rlandy | rascasoft: you're deploy rhos-13 on rdocloud??? | 14:49 |
| rlandy | pls so no | 14:49 |
| rlandy | say | 14:49 |
| rascasoft | rlandy, LOL no no | 14:49 |
| rlandy | ok - restart breathing | 14:49 |
| *** sanjayu__ has quit IRC | 14:50 | |
| rascasoft | rlandy, sorry if I made confusion, I just cited the only problem I have today with the undercloud | 14:50 |
| rlandy | rascasoft: running rhos-13 on ci-rhos, I upped the noba db sync timeout | 14:50 |
| rlandy | many times it still fails | 14:50 |
| *** ykarel has quit IRC | 14:54 | |
| rlandy | rascasoft: what release file are you using for rhos-13? the one from tripleo-envs? | 14:55 |
| rascasoft | rlandy, yes | 14:58 |
| rlandy | rdocloud back in business? | 14:58 |
| rascasoft | rlandy, do you have a link where are you failing? | 14:58 |
| rascasoft | rlandy, rdocloud is always in business, but I had to deal with so many failures these days, it's really hard | 14:59 |
| rlandy | rascasoft: I got one pass in many tries ... log links are down now though | 14:59 |
| rlandy | myoung: ^^ log server for internal? | 14:59 |
| rlandy | goes to rdocloud | 14:59 |
| myoung | rlandy: we can make a 1 line change and use internal logs, at least for rdo2 and osp0 jobs | 15:01 |
| * myoung reads scrollback | 15:01 | |
| * myoung is multitasking, tempest squad scrum starts now | 15:02 | |
| arxcruz | myoung: meeting now ? | 15:02 |
| *** sdoran has joined #oooq | 15:02 | |
| *** ykarel has joined #oooq | 15:03 | |
| *** udesale has quit IRC | 15:04 | |
| rlandy | myoung: not urgent | 15:05 |
| *** zul has quit IRC | 15:05 | |
| *** zul has joined #oooq | 15:09 | |
| *** kopecmartin has quit IRC | 15:16 | |
| rascasoft | rlandy, in the meantime I discovered what was making my overcloud deploys on master fail: https://bugs.launchpad.net/tripleo/+bug/1787910 | 15:17 |
| openstack | Launchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 15:17 |
| *** ykarel has quit IRC | 15:25 | |
| *** ykarel has joined #oooq | 15:29 | |
| rlandy | http://cistatus.tripleo.org/ - down with rdocloud? | 15:35 |
| rlandy | ugh - no job comparison logs | 15:36 |
| *** jtomasek has quit IRC | 15:38 | |
| *** ykarel_ has joined #oooq | 15:38 | |
| *** ykarel has quit IRC | 15:39 | |
| myoung | rlandy, rascasoft, want me to patch rdo2/osp0 to use internal logs until RDO cloud log server is back? | 15:44 |
| myoung | it's a quickie | 15:44 |
| rlandy | myoung: no - let's wait for rdocloud | 15:44 |
| myoung | ok | 15:44 |
| rlandy | I really need sova more than anything else right now :( | 15:44 |
| rlandy | trying to compare failures | 15:44 |
| myoung | sshnaidm: do we still have an instance running internally? | 15:45 |
| rascasoft | myoung, what rlandy says :) but on the other side I just created this https://code.engineering.redhat.com/gerrit/#/c/148775/ that fixes a couple of things for osp10, if you have time | 15:45 |
| myoung | re: logserver, if it does come to it, this is what we did last time this happened (dec 2016) http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/commit/jenkins/jobs/tripleo-quickstart?id=c5a942fab8940b4b1877ca889a68961f3300c914 | 15:46 |
| sshnaidm | myoung, no, but I will set up one.. | 15:46 |
| myoung | rascasoft: sure, can have a look | 15:46 |
| myoung | rascasoft: lgtm, +2, will merge with the test job you kicked off is completed | 15:52 |
| *** ykarel_ is now known as ykarel | 15:55 | |
| *** trown is now known as trown|lunch | 15:55 | |
| *** hamzy has quit IRC | 16:00 | |
| *** ykarel has quit IRC | 16:00 | |
| rlandy | panda|rover: rfolco: I logged all the errors (with comments) on https://trello.com/c/TVsZ3Ut6/877-clean-up-rdo-sf-legacy-code-after-zuulv3-migration | 16:07 |
| rlandy | we can compare results when we trigger again | 16:08 |
| *** ykarel has joined #oooq | 16:14 | |
| *** ykarel has quit IRC | 16:17 | |
| rlandy | panda|rover: need to run out to a doctor's appt - back in about 1hr+ | 16:19 |
| *** ccamacho has quit IRC | 16:19 | |
| *** rlandy is now known as rlandy|afk | 16:19 | |
| *** ykarel has joined #oooq | 16:25 | |
| *** ykarel_ has joined #oooq | 16:29 | |
| *** jfrancoa has quit IRC | 16:29 | |
| *** ykarel has quit IRC | 16:31 | |
| ssbarnea | does anyone knows how can I mention a storyboard ticker from inside the commit message? https://wiki.openstack.org/wiki/GitCommitMessages does not mention anything regarding storyboard. | 16:32 |
| tosky | Story: nnnnn | 16:33 |
| tosky | Task: mmmmm | 16:33 |
| *** jbadiapa has quit IRC | 16:35 | |
| *** agopi has quit IRC | 16:40 | |
| *** agopi has joined #oooq | 16:42 | |
| myoung | CI squad: http://lists.openstack.org/pipermail/openstack-dev/2018-August/134030.html | 16:49 |
| panda|rover | rlandy|afk: rfolco updated dependencies, I added a compatibility layer that transforms /etc/nodepool information in what we consume now, I hope it works | 16:50 |
| *** ykarel_ has quit IRC | 16:53 | |
| *** ykarel has joined #oooq | 16:55 | |
| *** ykarel has quit IRC | 16:55 | |
| *** ccamacho has joined #oooq | 16:58 | |
| *** ykarel has joined #oooq | 17:01 | |
| *** stack has joined #oooq | 17:03 | |
| *** dtantsur is now known as dtantsur|afk | 17:04 | |
| *** ykarel_ has joined #oooq | 17:06 | |
| *** ykarel has quit IRC | 17:08 | |
| *** agopi has quit IRC | 17:11 | |
| *** trown|lunch is now known as trown | 17:13 | |
| *** gkadam has joined #oooq | 17:15 | |
| myoung | ssbarnea, panda|rover, marios|rover, looking at rdo2, looks like most of the slaves are offline...is this known / tracked yet? | 17:22 |
| myoung | ^^ https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/computer/ | 17:22 |
| *** skramaja has quit IRC | 17:23 | |
| *** ccamacho has quit IRC | 17:24 | |
| *** hamzy has joined #oooq | 17:28 | |
| *** ykarel_ is now known as ykarel|away | 17:32 | |
| *** holser_ has quit IRC | 17:35 | |
| *** ykarel|away has quit IRC | 17:36 | |
| *** dmellado has joined #oooq | 17:44 | |
| *** agopi has joined #oooq | 17:48 | |
| rfolco | panda|rover, I don't understand why we need to read legacy /etc/nodepool (backward compat) if we are removing the pre.ymal that populates it? | 18:10 |
| rfolco | panda|rover, any job still populating /etc/nodepool ? | 18:11 |
| *** gkadam has quit IRC | 18:17 | |
| *** rlandy|afk is now known as rlandy | 19:05 | |
| rlandy | panda|rover: awesome thanks - will watch the jobs now | 19:06 |
| rlandy | rfolco: we need the compatibility layer as we have to merge the depends on changes first | 19:06 |
| rlandy | which will break before we merge the major review | 19:06 |
| rlandy | 2018-08-30 19:01:15.659341 | primary | to see the full traceback, use -vvv | 19:12 |
| rlandy | 2018-08-30 19:01:15.725697 | primary | /home/zuul/workspace/devstack-gate/functions.sh: line 150: /home/zuul/workspace/logs/reproduce.sh: No such file or directory | 19:12 |
| rlandy | ssbarnea, ^^ you know about the above error? | 19:14 |
| rlandy | panda|rover: ? | 19:14 |
| rlandy | rfolco: ^^?? | 19:25 |
| rlandy | anyone home? | 19:25 |
| rfolco | rlandy, devstack-gate... oh jeez | 19:26 |
| rlandy | rfolco: you know what broke this? | 19:26 |
| rfolco | no | 19:26 |
| rlandy | ok - will have to dig in here | 19:27 |
| rlandy | something merged | 19:27 |
| rfolco | legacy jobs still use it... | 19:27 |
| rlandy | what deleted it? | 19:30 |
| rlandy | https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L1358 | 19:35 |
| rlandy | nothing should have deleted that | 19:35 |
| *** hamzy has quit IRC | 19:54 | |
| *** hamzy has joined #oooq | 19:55 | |
| rlandy | 2018-08-30 19:33:47.759662 | primary | ERROR! Unexpected Exception: No module named manager | 20:02 |
| rlandy | rfolco: ^^ real problem I think | 20:02 |
| *** centos has joined #oooq | 20:04 | |
| *** centos is now known as Guest58453 | 20:04 | |
| rfolco | rlandy, fatal? | 20:04 |
| rlandy | yes | 20:04 |
| rlandy | https://github.com/openstack/tripleo-quickstart/commit/60f9a8fdf14f3f6f023a8616107bea7b448fddd6 | 20:04 |
| rfolco | revert? | 20:06 |
| rlandy | idk | 20:07 |
| rlandy | I don;t see it on upstream | 20:08 |
| rlandy | https://review.openstack.org/#/c/587371/ | 20:09 |
| rlandy | merged 30 hours ago | 20:10 |
| rlandy | let's try a revert | 20:10 |
| rlandy | rfolco: shoot - revert didn;t work ERROR! Unexpected Exception: No module named manager | 20:20 |
| rfolco | rlandy, please paste the link I can try to help finding root cause | 20:21 |
| rlandy | https://logs.rdoproject.org/39/598339/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/899cc92/job-output.txt.gz#_2018-08-30_20_19_35_675611 | 20:21 |
| rlandy | rfolco:I'm comparing to previous jobs | 20:21 |
| rlandy | 2018-08-30 20:19:29.238903 | tripleo-ovb-centos-7 | ara 0.16.0 has requirement ansible>=2.4.5.0, but you'll have ansible 2.3.2.0 which is incompatible. | 20:25 |
| rlandy | cat: /etc/nodepool/primary_node_private: No such file or directory seems ok | 20:26 |
| rlandy | rfolco: this is the real problem ... | 20:27 |
| rlandy | https://logs.rdoproject.org/39/598339/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/899cc92/job-output.txt.gz#_2018-08-30_20_19_36_298102 | 20:27 |
| rlandy | panda|rover's patch is also broken | 20:32 |
| rlandy | but will fix that later | 20:33 |
| rfolco | from ansible.config.manager import find_ini_config_file | 20:36 |
| rfolco | rlandy, but this is ansible error | 20:36 |
| rlandy | rfolco: ^^ what are pointing to here? | 20:38 |
| rlandy | ssbarnea: you around? | 20:38 |
| ssbarnea | rlandy: on mobile, ready to go to sleep. anything urgent? | 20:40 |
| rlandy | ssbarnea: rdocloud jobs are broken | 20:40 |
| rlandy | https://logs.rdoproject.org/39/598339/2/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/c9efb7c/job-output.txt.gz#_2018-08-30_20_37_06_777029 | 20:40 |
| rlandy | we are looking at this error: | 20:40 |
| rlandy | 2018-08-30 20:37:08.606139 | tripleo-ovb-centos-7 | ERROR! Unexpected Exception: No module named manager | 20:40 |
| rlandy | I tried a revert of your patch | 20:41 |
| rlandy | but it didn't help :( | 20:41 |
| rlandy | ssbarnea: did you test on rdocloud? | 20:42 |
| rlandy | the patch does not show those results | 20:42 |
| ssbarnea | yep, it is ansible but not the minor version bump, is likely using ansibke smaller than 2.4.0 | 20:42 |
| rlandy | I am comparing with | 20:43 |
| rlandy | http://logs.rdoproject.org/25/561725/10/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/172eceb/job-output.txt.gz | 20:43 |
| ssbarnea | ansible moved manager module at some point, i remember a fix in infrared for dealing with that. | 20:43 |
| rlandy | ssbarnea: ^^ see the diff> | 20:43 |
| rlandy | 2018-08-30 08:40:16.356658 | tripleo-ovb-centos-7 | [WARNING]: No hosts matched, nothing to do | 20:43 |
| rlandy | 2018-08-30 08:40:17.755555 | tripleo-ovb-centos-7 | localhost | SUCCESS => {\ | 20:43 |
| ssbarnea | let me go to computer | 20:43 |
| rlandy | k | 20:43 |
| rfolco | rlandy, thats it... python -c "from ansible.config.manager import find_ini_config_file" | 20:48 |
| rfolco | --> ansible 2.7 works | 20:48 |
| rfolco | rlandy, same for 2.5 doesn't | 20:48 |
| rfolco | just tested in a venv | 20:48 |
| rfolco | brb | 20:48 |
| ssbarnea | rlandy: cannot blame my bump from 2.5.3 to 2.5.8 when this failure is caused by the fact that version installed is ansible-2.3.2.0 ! which is clearly the source of the problem. | 20:49 |
| rlandy | ssbarnea: yeah - I know the revert did not work for that | 20:49 |
| ssbarnea | rlandy: i know the manager issue, as we encounter it downstream, about 3-4 months ago. | 20:50 |
| rlandy | what happened to versions> | 20:50 |
| rlandy | 2018-08-30 20:36:59.441068 | tripleo-ovb-centos-7 | ara 0.16.0 has requirement ansible>=2.4.5.0, but you'll have ansible 2.3.2.0 which is incompatible. | 20:50 |
| rlandy | pip upgrade ansible | 20:51 |
| ssbarnea | when we migrated from 2.4 to 2.5 (or 2.3 to 2.4 not sure). is part of the breaking changes in ansible release notes. | 20:51 |
| rlandy | but this breaks from yesterday to today | 20:52 |
| ssbarnea | well, pip deps are known to cause problems, this is why I always do a "pip check" after running pip install, to stop the build if there are conflicts reported. | 20:52 |
| rlandy | what upgrades then? | 20:52 |
| rlandy | was not quickstart | 20:52 |
| ssbarnea | if fact I think is what downgrades it, not what upgrades it. | 20:53 |
| ssbarnea | i think the pip install command is part of this script https://logs.rdoproject.org/39/598339/2/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/c9efb7c/job-output.txt.gz#_2018-08-30_20_36_13_776888 | 20:54 |
| ssbarnea | rlandy: found: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh#L31 | 20:56 |
| ssbarnea | if we define a different version we can avoid that issue. | 20:56 |
| rlandy | weird this was a year ago | 20:59 |
| ssbarnea | is the first time i see this repo, but this hardcoded line makes me wonder who decides which ansible we need, if each repo has its own version we will never be able make them work. | 20:59 |
| *** hamzy has quit IRC | 20:59 | |
| ssbarnea | 2.3 is VERY old | 21:00 |
| rlandy | only of ANSIBLE_VERSION is not defined | 21:00 |
| rlandy | did we remove that definition? | 21:00 |
| rlandy | I am just trying to mail down what changed | 21:00 |
| *** trown is now known as trown|outtypewww | 21:03 | |
| ssbarnea | rlandy: sorted! just wait a little bit others are working to fix it. See https://review.openstack.org/#/c/591527/ - | 21:05 |
| rlandy | ssbarnea: cool | 21:05 |
| rlandy | ok - now to fix panda|rover's patch | 21:06 |
| ssbarnea | i will go to sleep, is 10pm here and I woke at 6am. CC me on any reviews or even email me, i will takeover early morning. ok? | 21:08 |
| ssbarnea | even so, my ability to help is limited to +/-1 anyway ;) | 21:09 |
| rlandy | ssbarnea: good night - we can handle it from here | 21:10 |
| rlandy | sorry I accused your patch :( | 21:11 |
| rlandy | it was my first guess | 21:11 |
| *** holser_ has joined #oooq | 21:19 | |
| *** dmellado has quit IRC | 21:31 | |
| *** dmellado has joined #oooq | 21:31 | |
| *** holser_ has quit IRC | 21:37 | |
| rlandy | rfolco: getting there | 21:40 |
| *** agopi has quit IRC | 21:40 | |
| rlandy | fixed panda|rover patch | 21:41 |
| rlandy | waiting on devstack gate fix | 21:41 |
| *** holser_ has joined #oooq | 21:42 | |
| *** myoung has quit IRC | 21:47 | |
| *** tosky has quit IRC | 22:23 | |
| *** holser_ has quit IRC | 22:30 | |
| *** sshnaidm is now known as sshnaidm|off | 22:35 | |
| *** dbecker has quit IRC | 22:37 | |
| *** dbecker has joined #oooq | 23:06 | |
| *** agopi has joined #oooq | 23:12 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!