| dtrainor_ | I'm trying to deploy using --release rhos-14 with an appropriate release file. the deployment keeps failing on account of not being able to find a suitable ntp server. i'm specifying an 'undercloud_ntp_servers' properly in my config/release/rhos-14.yml file. though i can confirm that the release file seems to be working properly, the undercloud_ntp_servers setting is not taking effect | 00:06 |
|---|---|---|
| dtrainor_ | the failure is using pool.ntp.org | 00:06 |
| dtrainor_ | is there a more appropriate place to configure 'undercloud_ntp_servers'? | 00:06 |
| rlandy | 2018-08-29 00:05:23.353909 | primary | fatal: [undercloud]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: \"hostvars['primary']\" is undefined"} | 00:33 |
| rlandy | will be back later to fix ... | 00:33 |
| *** rlandy is now known as rlandy|bbl | 00:33 | |
| *** sdoran has quit IRC | 01:19 | |
| *** dtantsur|afk has quit IRC | 01:19 | |
| *** rlandy|bbl is now known as rlandy | 02:13 | |
| *** hamzy has quit IRC | 02:17 | |
| *** hamzy has joined #oooq | 02:18 | |
| *** openstack has joined #oooq | 02:51 | |
| *** ChanServ sets mode: +o openstack | 02:52 | |
| *** openstackstatus has joined #oooq | 02:59 | |
| *** ChanServ sets mode: +v openstackstatus | 02:59 | |
| *** rlandy has quit IRC | 03:30 | |
| *** gkadam has joined #oooq | 03:39 | |
| *** udesale has joined #oooq | 03:59 | |
| *** saneax has joined #oooq | 04:16 | |
| *** myoung is now known as myoung|pto | 04:25 | |
| *** ykarel has joined #oooq | 05:28 | |
| *** ratailor has joined #oooq | 05:35 | |
| *** ykarel has quit IRC | 05:42 | |
| *** ykarel has joined #oooq | 05:52 | |
| ykarel | cool, rocky being promoted: 2018-08-29 05:16:39,897 30524 INFO promoter Promoting the container images for dlrn hash 3771eb0202782879a780df444f0b5e32446f4712 on rocky to current-tripleo | 05:53 |
| ykarel | master still not moved, as rdo repo is not consistent due to glance FTBFS:- https://review.rdoproject.org/r/#/c/16010/ | 05:59 |
| *** ccamacho has joined #oooq | 06:03 | |
| *** udesale has quit IRC | 06:07 | |
| *** jfrancoa has joined #oooq | 06:15 | |
| *** kopecmartin has joined #oooq | 06:18 | |
| *** ykarel has quit IRC | 06:23 | |
| *** udesale has joined #oooq | 06:55 | |
| *** ykarel has joined #oooq | 06:55 | |
| *** apetrich has joined #oooq | 07:02 | |
| *** ykarel has quit IRC | 07:05 | |
| *** ykarel has joined #oooq | 07:10 | |
| *** ykarel has quit IRC | 07:10 | |
| *** ykarel has joined #oooq | 07:10 | |
| *** ykarel has quit IRC | 07:11 | |
| *** ykarel has joined #oooq | 07:12 | |
| *** jtomasek has joined #oooq | 07:12 | |
| *** ykarel has quit IRC | 07:13 | |
| *** ratailor_ has joined #oooq | 07:19 | |
| *** ratailor has quit IRC | 07:20 | |
| rascasoft | wow finding two reviews merged during the night is a good feeling :) | 07:21 |
| ssbarnea | we need wes fix reviewed and merged asap to unblock queens jobs: https://review.openstack.org/#/c/597141/ | 07:22 |
| *** udesale has quit IRC | 07:31 | |
| *** tosky has joined #oooq | 07:38 | |
| *** ykarel has joined #oooq | 07:49 | |
| *** ccamacho has quit IRC | 08:13 | |
| *** ccamacho has joined #oooq | 08:14 | |
| *** ykarel is now known as ykarel|away | 08:27 | |
| *** ykarel|away has quit IRC | 08:29 | |
| rascasoft | ssbarnea, I still wonder why this is not merged, it has everything it needs | 08:34 |
| ssbarnea | rascasoft: if you refer to the one i posted, it just got the +W so it should merge in ~2h from now, hopefully. | 08:35 |
| rascasoft | ssbarnea, yeah you're right | 08:36 |
| rascasoft | ssbarnea, I was thinking about the other one we were talking yesterday | 08:36 |
| ssbarnea | rascasoft: if I remember well it was affected by this one. | 08:38 |
| rascasoft | ssbarnea, I think so, yes. In any case I just discovered a very very funny thing about introspection, which is not working anymore in master | 08:38 |
| rascasoft | ssbarnea, we still rely on the ironic command and now it is deprecated | 08:39 |
| ssbarnea | rascasoft: i am afraid i do not know the details. | 08:40 |
| rascasoft | ssbarnea, no worries I was just saying :) I'm preparing a review | 08:40 |
| *** dtantsur has joined #oooq | 08:42 | |
| *** ccamacho has quit IRC | 08:55 | |
| *** ccamacho has joined #oooq | 08:56 | |
| *** ykarel has joined #oooq | 08:56 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:59 | |
| rascasoft | ssbarnea, this is what I was talking about https://review.openstack.org/#/c/597432/ | 09:27 |
| rascasoft | sshnaidm, this ^^^ is really relevant for baremetal | 09:27 |
| rascasoft | I'm testing it right now | 09:27 |
| sshnaidm | rascasoft, yeah, seems reasonable. Just need to replace tabs by spaces | 09:30 |
| *** holser_ has joined #oooq | 09:30 | |
| ssbarnea | sshnaidm: is "openstack baremetal" command available for all supported openstack older versions? | 09:33 |
| sshnaidm | ssbarnea, we support from newton I think, and imho newton and ocata will be EOLed soon (if not already) | 09:34 |
| sshnaidm | ssbarnea, so yes | 09:34 |
| *** ykarel is now known as ykarel|away | 09:36 | |
| ssbarnea | sshnaidm: ok, had to ask because I still remember when tripleo changes broke downstream jobs as those do usually have to support some ancient versions. | 09:36 |
| sshnaidm | ssbarnea, yeah, but you're not in downstream anymore, so you can start ignoring such things :D | 09:37 |
| ssbarnea | haha :D | 09:37 |
| ssbarnea | sshnaidm: it seems that my rdo cloud account does not have a baremetal endpoint. do you know one test cloud that does? | 09:41 |
| sshnaidm | ssbarnea, what is baremetal endpoint? | 09:42 |
| ssbarnea | i just wanted to play with baremetal sub-command and discovered that i don't have something to play with. | 09:42 |
| ssbarnea | public endpoint for baremetal service in regionOne region not found | 09:42 |
| sshnaidm | ssbarnea, you can't play with baremetals of rdo cloud, it's only for admins | 09:42 |
| sshnaidm | ssbarnea, you need to install OVB for that | 09:43 |
| sshnaidm | ssbarnea, for example some OVB job | 09:43 |
| *** ykarel|away has quit IRC | 09:43 | |
| sshnaidm | and play wit baremetal on ovb | 09:43 |
| ssbarnea | sshnaidm: ok. so only via my own deployment. | 09:44 |
| *** udesale has joined #oooq | 09:52 | |
| *** ykarel has joined #oooq | 09:57 | |
| rascasoft | ssbarnea, sshnaidm, newton is already EOL, but in any case it supports the openstack command, so we're good | 09:58 |
| rascasoft | sshnaidm, that joke about downstream was not good man :D :D | 09:58 |
| sshnaidm | rascasoft, it's worse - not a joke :D | 10:06 |
| sshnaidm | rascasoft, I'm kidding, of course we care about downstream | 10:07 |
| sshnaidm | once a half year | 10:07 |
| rascasoft | sshnaidm, lol one day or another I'll stop living in this middlearth between the two worlds and start being fine again :D | 10:07 |
| sshnaidm | rascasoft, join us in upstream as ssbarnea did, he knows :) | 10:08 |
| rascasoft | sshnaidm, you have cookies? | 10:08 |
| sshnaidm | rascasoft, only in firefox | 10:09 |
| rascasoft | LOL | 10:10 |
| *** ratailor_ has quit IRC | 10:23 | |
| *** saneax has quit IRC | 10:33 | |
| rfolco | sshnaidm, panda: https://review.openstack.org/#/c/589448/ looks good, needs a final pass/review | 10:48 |
| sshnaidm | rfolco, will review today | 10:50 |
| rfolco | sshnaidm, thank you | 10:51 |
| *** d0ugal has quit IRC | 11:03 | |
| *** udesale has quit IRC | 11:04 | |
| rascasoft | we need to do something about the pip cache not cleaning | 11:06 |
| ykarel | rascasoft, is that jenkins issue, i remember one in the past which was workaround some how | 11:07 |
| sshnaidm | ssbarnea, are you ruck? | 11:07 |
| sshnaidm | who is ruck/rover today? | 11:08 |
| ykarel | hmm good to set in topic, if not in nick, othewise by default i ping sshnaidm :) | 11:09 |
| rascasoft | ykarel, yep it is the jenkins issue, I can't count how much time I lost for this problem | 11:09 |
| panda | I offered to help, not sure I am the official ruck / rover today | 11:09 |
| rascasoft | LOL sshnaidm is your best friend | 11:09 |
| rascasoft | who you gonna call?! | 11:10 |
| sshnaidm | arxcruz, seems like you're only channel op here :) | 11:12 |
| sshnaidm | arxcruz, need to remove "rdo cloud is down" from topic.. | 11:12 |
| *** ChanServ sets mode: +o panda | 11:13 | |
| ykarel | rascasoft, so is it solved now ? would u like me to look for that earlier bug | 11:13 |
| *** panda changes topic to "Docs: https://docs.openstack.org/tripleo-quickstart/latest/ || RUCK is ssbarnea, ROVER is unknown" | 11:13 | |
| panda | ssbarnea: are you going to attend the program meeting ? | 11:14 |
| *** panda sets mode: -o panda | 11:14 | |
| ssbarnea | oops | 11:15 |
| ssbarnea | panda: yes, but i have a clash with 1:1 with wes, but based on his pto status, i would assume that I will attend the platform program meeting. | 11:16 |
| *** d0ugal has joined #oooq | 11:18 | |
| panda | ssbarnea: wes is out until Tuesday | 11:19 |
| ssbarnea | panda: it seems that i have lots of timeouts on ocata, i need to look now on them.... | 11:20 |
| ssbarnea | out of curiosity, do any of you have some secret tool for pointing the failure point in zuul log, i feel that i waste too much time scrolling to find the error line. | 11:21 |
| rascasoft | ykarel, no it isn't solved, I need to remember to remove .cache dir on jenkins slave before launching each job | 11:32 |
| ssbarnea | panda: can you help me with the error about missing bm-deploy-kernel ? http://logs.openstack.org/91/564291/20/check/tripleo-ci-centos-7-scenario001-multinode-oooq/40fb802/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 11:33 |
| ykarel | rascasoft, so possibly u can try the workaround mentioned here: https://bugs.launchpad.net/tripleo/+bug/1772460 | 11:34 |
| openstack | Launchpad bug 1772460 in tripleo "rdo2: BM jobs failing b/c concurrent pip installs are failing due to sharing pip cache" [Critical,Fix released] - Assigned to Matt Young (halcyondude) | 11:34 |
| sshnaidm | rfolco, the only difference I found: in changed version the file featureset-multinode-common.yml is taken from sources - /home/zuul/src/git.openstack.org/openstack/tripleo-quickstart and in original it's taken from workspace: /home/zuul/workspace/.quickstart | 11:36 |
| *** abishop has quit IRC | 11:36 | |
| sshnaidm | rfolco, but I think it doesn't matter | 11:36 |
| rfolco | sshnaidm, exactly what I commented out in the card :) | 11:37 |
| sshnaidm | rfolco, cool, afaik we don't change it.. | 11:37 |
| sshnaidm | rfolco, so +2 | 11:37 |
| rfolco | sshnaidm, I don't understand why still using local_working_dir for some configs... master.yml for example.... | 11:38 |
| sshnaidm | rfolco, seems like we need to decide where to take them from - one place | 11:38 |
| sshnaidm | rfolco, imho sources are better, it makes us less dependent on localhost and we shouldn't change these configs.. | 11:39 |
| rfolco | sshnaidm, zuulv3 resolves cross dependencies among repos... we don't need to rely on reuqirements for gating projects IMHO | 11:40 |
| rfolco | but this is a separate work | 11:40 |
| rfolco | tech debt | 11:40 |
| sshnaidm | panda, he said tech debt ^^ | 11:40 |
| rfolco | lolz | 11:40 |
| * rfolco hides | 11:41 | |
| *** arxcruz sets mode: +o sshnaidm | 11:47 | |
| arxcruz | sshnaidm: now you got the power :) | 11:47 |
| ssbarnea | panda: sshnaidm : if building openstack-tripleo-heat-templates fails with DLRN complaining about missing python-d2to1 -- where should I file a bug? | 11:47 |
| sshnaidm | ssbarnea, I think we talked about it yesterday in tripleo meeting | 11:48 |
| sshnaidm | ykarel, do you remember what is the problem? ^^ | 11:48 |
| ssbarnea | i remember something about it but i missed to see any action items | 11:49 |
| sshnaidm | ssbarnea, I think jpena frm #rdo was on it, worth to ask him | 11:49 |
| ssbarnea | thanks | 11:49 |
| ykarel | sshnaidm, probably missing build-deps | 11:50 |
| ykarel | one recently resolved | 11:50 |
| ykarel | sshnaidm, ssbarnea https://review.openstack.org/#/c/597154/ | 11:51 |
| ykarel | is it still failing? where? | 11:51 |
| ssbarnea | ykarel: http://logs.openstack.org/96/596696/3/check/tripleo-ci-centos-7-undercloud-containers/106ba33/logs/delorean_logs/64/69/6469e195992978fc59c8f3ef917e6662019ed673_dev/rpmbuild.log.txt.gz | 11:52 |
| *** ssbarnea|ruck has joined #oooq | 11:54 | |
| ykarel | sshnaidm, so the failure is from 27th, fix merged today morning | 11:55 |
| ykarel | ssbarnea, ^^ | 11:55 |
| ykarel | sshnaidm, so if u see the current run, job moved ahead http://zuul.openstack.org/stream.html?uuid=f81d3e0e6b44471c9c3f57b9b939aa5b&logfile=console.log | 11:57 |
| ykarel | sorry, | 11:57 |
| ykarel | meant for ssbarnea ^^ | 11:57 |
| ssbarnea | ok, so I will just want for the check to finish. | 11:58 |
| ykarel | yes | 11:58 |
| *** ssbarnea|ruck has quit IRC | 12:03 | |
| rascasoft | ykarel, the workaround is operative, but it does not prevent the problem to happen | 12:04 |
| ykarel | hmm no idea about that | 12:06 |
| *** abishop has joined #oooq | 12:07 | |
| *** abishop has left #oooq | 12:07 | |
| *** ccamacho has quit IRC | 12:07 | |
| *** ccamacho has joined #oooq | 12:09 | |
| *** udesale has joined #oooq | 12:14 | |
| *** udesale has quit IRC | 12:16 | |
| *** udesale has joined #oooq | 12:19 | |
| *** udesale has quit IRC | 12:19 | |
| *** ykarel_ has joined #oooq | 12:20 | |
| *** ykarel has quit IRC | 12:22 | |
| *** trown|outtypewww is now known as trown | 12:25 | |
| *** udesale has joined #oooq | 12:28 | |
| *** udesale has quit IRC | 12:28 | |
| *** sshnaidm is now known as sshnaidm|afk | 12:28 | |
| *** rlandy has joined #oooq | 12:29 | |
| *** udesale has joined #oooq | 12:31 | |
| *** ykarel_ is now known as ykarel | 12:36 | |
| rlandy | panda: hello | 12:38 |
| rlandy | panda: wrt https://review.openstack.org/#/c/596422/ - there is one more change need to vxlan networking that I wanted to run by you | 12:38 |
| *** gkadam has quit IRC | 12:41 | |
| panda | rlandy: shoot | 12:43 |
| rlandy | panda: vxlan networking is only used by legacy | 12:45 |
| rlandy | that is why ovb passes and multinode fails legacy | 12:45 |
| rlandy | the issue is simple though | 12:45 |
| rlandy | we nee to define primary node | 12:45 |
| rlandy | https://review.openstack.org/#/c/596906/3/roles/undercloud-setup/templates/vxlan_networking.sh.j2 | 12:46 |
| rlandy | panda: I wanted to add it to the /tmp/subnodes file | 12:46 |
| rlandy | https://review.rdoproject.org/r/#/c/16013/4/playbooks/legacy/pre.yaml | 12:46 |
| ssbarnea | panda: rlandy please have a look at https://review.openstack.org/#/c/596381/ -- is related to timeouts on updates. i had to add an extra feature based on yuri feedback. | 12:46 |
| rlandy | add a switch section to subnodes | 12:46 |
| rlandy | it would be one value though | 12:47 |
| rlandy | panda: then - we would have access to switch ip like the subnodes | 12:49 |
| rlandy | I'd like to use the same file so we are not passing more extra-vars | 12:49 |
| rlandy | panda:^^ thoughts? | 12:49 |
| rlandy | ssbarnea: openstack/tripleo-upgrade should be approved by upgrades team | 12:51 |
| rlandy | is marios back? | 12:52 |
| panda | rlandy: ok, maybe we can rename the file to deployed_nodes.yaml. THen use a dict with two lists. primaery: and subnodes: | 12:52 |
| rlandy | panda: yes - my thought exactly | 12:53 |
| rlandy | panda: what I was not sure about is ... | 12:53 |
| rlandy | is it safe to assume that there will always only be one primary node? | 12:53 |
| rlandy | considering hrybacki's work | 12:53 |
| rlandy | like setting switch string or list | 12:54 |
| panda | rlandy: uhm , not sure what changes with harry's work, they are adding a node to the mix. The primary for us is the undercloud, if there is not undercloud, we are not testing anything | 12:55 |
| rlandy | panda: when we working with OVB we consider it an extra undercloud node ... but for the moment, let's ignore that | 12:56 |
| rlandy | either way ... | 12:56 |
| rlandy | "THen use a dict with two lists. primary: and subnodes:" is fine | 12:57 |
| rlandy | we will need to make the change in https://review.rdoproject.org/r/#/c/16013/2..4/playbooks/legacy/pre.yaml | 12:57 |
| rlandy | and in https://review.openstack.org/#/c/596422/27/playbooks/tripleo-ci/run-v3.yaml | 12:57 |
| panda | rlandy: ok, I can change the 422 | 12:57 |
| ssbarnea | panda: i know that upgrades should approve in the end, but I would also appreciate your input, to get an idea if the approach looks ok or not. | 12:57 |
| rlandy | panda: ok - when you have done that, I will update https://review.rdoproject.org/r/#/c/16013/2..4/playbooks/legacy/pre.yaml and https://review.openstack.org/#/c/596906/3/roles/undercloud-setup/templates/vxlan_networking.sh.j2 and recheck to test | 12:58 |
| *** ccamacho has quit IRC | 12:58 | |
| *** ccamacho has joined #oooq | 12:59 | |
| *** sshnaidm|afk is now known as sshnaidm | 13:02 | |
| panda | rlandy: ps uploaded | 13:02 |
| panda | rlandy: mmhh forgot the legacy scripts | 13:03 |
| *** dtantsur is now known as dtantsur|bbl | 13:04 | |
| panda | ok, fixed legacy scripts | 13:05 |
| panda | ssbarnea: taking a look | 13:10 |
| panda | ssbarnea: I think we should do the same thing on every place we are currently using a "ignore_errors" | 13:15 |
| rlandy | panda: k - updating other reviews | 13:16 |
| *** saneax has joined #oooq | 13:18 | |
| rlandy | panda: for vxlan networking ... now we just set ... | 13:20 |
| rlandy | primary_node="{{ primary }}" | 13:20 |
| rlandy | oh and one more ... | 13:22 |
| rlandy | https://review.openstack.org/#/c/596887/3/roles/tripleo-inventory/tasks/inventory.yml | 13:22 |
| *** sanjayu_ has joined #oooq | 13:24 | |
| *** saneax has quit IRC | 13:26 | |
| rlandy | panda: pls check https://review.openstack.org/#/c/596887/4/roles/tripleo-inventory/tasks/inventory.yml and https://review.openstack.org/#/c/596906/5/roles/undercloud-setup/templates/vxlan_networking.sh.j2 | 13:34 |
| rlandy | before I recheck | 13:34 |
| panda | rlandy: mhh, I'm not sure you need to add the primary node to the inventory | 13:42 |
| panda | rlandy: it's the undercloud, should already be on the inventory in some way | 13:42 |
| panda | rlandy: we didn't add it before | 13:42 |
| panda | rlandy: or you want to add the alias "primary" to the inventory | 13:42 |
| panda | rlandy: also, now that I see it "primary" is really generic name for a variable, maybe we should rename it to primary_node | 13:43 |
| rlandy | panda: idk - just seemed more consistent to add it | 13:44 |
| rlandy | ok if you want to change the name | 13:44 |
| rlandy | switch to alias | 13:44 |
| panda | rlandy: renamed | 13:48 |
| rlandy | panda: ok - need to update the other reviews | 13:50 |
| rlandy | what's the decision on the inventory? | 13:50 |
| rlandy | take it out? | 13:50 |
| rlandy | panda: and it I remove it - what to use here? https://review.openstack.org/#/c/596906/5/roles/undercloud-setup/templates/vxlan_networking.sh.j2? | 13:52 |
| ssbarnea | rlandy: panda : wes raised this two days ago and now it passed, about timeouts collecting,... worth merging. | 13:53 |
| ssbarnea | sshnaidm: ^^ | 13:53 |
| panda | rlandy: I think we can take it out. It wasn't there before, and you dont' even need an alias, I think you're using just the ip. Indeed, I think the vxlan role couls use the existing inventory, but let's not change too many things ... | 13:53 |
| sshnaidm | ssbarnea, ?? | 13:54 |
| rlandy | panda: so for https://review.openstack.org/#/c/596906/5/roles/undercloud-setup/templates/vxlan_networking.sh.j2, just remove line 175? | 13:54 |
| panda | rlandy: no, use {{ primary_node }} there. I *think* we could just use hostvars["undercloud"]['ip'] or something similar. I just suggested there is no need to add the primary node to the inventory, since that part is only for the subnodes specifically | 13:56 |
| *** dtantsur|bbl is now known as dtantsur | 13:58 | |
| rlandy | panda; I removed primary from the inventory. wrt vxlan networking, I did this ... https://review.openstack.org/#/c/596906/6/roles/undercloud-setup/templates/vxlan_networking.sh.j2 | 13:59 |
| rlandy | per your suggestion | 13:59 |
| rlandy | but hostvars["undercloud"]['ip' would have made more sense | 14:00 |
| rlandy | anyways, I think we are talking at cross purposes, pls comment on the reviews or edit them | 14:00 |
| panda | rlandy: ok | 14:03 |
| ssbarnea | rlandy: oops, missed to paste the link https://review.openstack.org/#/c/596841/ | 14:04 |
| rlandy | panda: pls let me know when we are good to go on the those reviews and we can recheck/watch | 14:07 |
| *** ykarel is now known as ykarel|afk | 14:11 | |
| panda | rlandy: 1 comment in https://review.openstack.org/596906 | 14:12 |
| panda | rlandy: 1 comment in https://review.rdoproject.org/api/16013 | 14:12 |
| panda | rlandy: the other seems good | 14:13 |
| rlandy | k - looking | 14:13 |
| rlandy | ah good catch | 14:13 |
| *** ykarel|afk has quit IRC | 14:15 | |
| ssbarnea | panda: i found few duplicate keys in feature sets which definetly are causing bugs, can you help me distinguish the correct keys? look at https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset021.yml#L103 | 14:20 |
| ssbarnea | which tempest_format is the correct one, first or second? | 14:20 |
| rascasoft | rlandy, hey good morning! | 14:23 |
| panda | ssbarnea: arxcruz is in the tempest squad. I think they are both valid, but they should be merged. One controls the format according to branch, the other according to feature | 14:23 |
| * arxcruz checking | 14:23 | |
| panda | ssbarnea: arxcruz it is indeed confusing | 14:23 |
| rascasoft | rlandy, I'm testing https://review.openstack.org/#/c/597432/ and I think you are the one most interested in it | 14:24 |
| arxcruz | wow... | 14:24 |
| arxcruz | ssbarnea: we should merge it | 14:25 |
| arxcruz | because containers are only on rocky and onwards | 14:25 |
| arxcruz | actually, checking for undercloud containerized is enough right now | 14:26 |
| arxcruz | panda: undercloud containerized is from rocky and onwards or queens ? | 14:26 |
| rlandy | panda: something like ... | 14:26 |
| rlandy | {% if subnodes.1 is defined %} | 14:26 |
| rlandy | {% set sub_nodes = sub_nodes + '' + "{{ subnodes.1 }}" %} | 14:26 |
| rlandy | rascasoft:looking | 14:27 |
| ssbarnea | arxcruz: fixing it now, a will cc you on review. | 14:27 |
| arxcruz | ssbarnea: but now i'm in doubt when we started to have undercloud containerized | 14:27 |
| arxcruz | because if was from rocky we can keep only the last one | 14:27 |
| arxcruz | if not, we need to merge it | 14:27 |
| panda | arxcruz: I think it's rocky | 14:28 |
| arxcruz | if it's rocky, we can keep only the last one | 14:28 |
| panda | arxcruz: it's difficult to remember, I think we introduced them right before queens was released | 14:29 |
| rlandy | rascasoft: is ocata officially dead? | 14:30 |
| panda | rlandy: that subnodes variable in vxlan_networking is bash. I think you can just use sub_nodes+=" {{ subnodes.1}}" | 14:30 |
| * rlandy is concerned about long term support | 14:30 | |
| rascasoft | rlandy, no I'm not aware of it | 14:30 |
| rascasoft | rlandy, but it should not cound | 14:31 |
| rascasoft | *count | 14:31 |
| rascasoft | I mean, openstack commands are good also in ocata... or not? | 14:31 |
| rlandy | confiminf gate runs | 14:31 |
| *** dtrainor_ has quit IRC | 14:35 | |
| rlandy | rascasoft: we still run legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch gate - let's see what shakes out | 14:35 |
| *** dtrainor has joined #oooq | 14:35 | |
| dtrainor | is there a more appropriate place to configure 'undercloud_ntp_servers' that in a config/releases/ file? | 14:39 |
| dtrainor | *than in | 14:39 |
| dtrainor | doesn't look like that takes effect, the deploy times out on account of still trying to use the default ntp server of pool.ntp.org | 14:40 |
| rlandy | dtrainor: pls see https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-deploy/templates/undercloud.conf.j2#L95 | 14:41 |
| rlandy | you can set {{ undercloud_undercloud_ntp_servers }} in a config file | 14:42 |
| dtrainor | right, that's what I'm doing, including it in my release file of config/releases/rhos-14.yml. Is that not one of the parameters that the role file should supply? | 14:44 |
| rlandy | it should be included in some sort of env setting type of file | 14:53 |
| rlandy | but there is no technical reason why including it in a config/releases/rhos-14.yml should not work | 14:54 |
| rlandy | depending on how that is passed | 14:54 |
| *** dtrainor has quit IRC | 14:54 | |
| rlandy | is it passed as --extra-var @file | 14:54 |
| rlandy | or --release | 14:54 |
| *** dtrainor has joined #oooq | 14:54 | |
| rlandy | it is not supplied by a featureset as it is env specific | 14:55 |
| *** vinaykns has joined #oooq | 14:56 | |
| *** dtrainor has quit IRC | 14:56 | |
| *** dtrainor has joined #oooq | 14:57 | |
| *** ykarel has joined #oooq | 14:57 | |
| dtrainor | I see, thanks. | 14:58 |
| dtrainor | So what's the *right* way to do this? The proper, intended, proposed way of doing it correctly. | 14:58 |
| rlandy | can you paste your current command? | 15:01 |
| rlandy | here is an example of where we use it .. http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/environments/ci-rhos.yml#n54 | 15:02 |
| dtrainor | it's pretty darn basic, wasn't even including a --config, just letting the role specification do its thing | 15:04 |
| dtrainor | i guess my point is, this isn't in the docs either | 15:04 |
| dtrainor | i just wanted to make sure i was doing it the right way | 15:04 |
| dtrainor | so one should stuff whatever custom configuration parameters they wish, inside of a config/environment/? is there any functional difference between a config file and an environment file or do they just happen to be in different directories? | 15:05 |
| ykarel | so time to add fs037 back to rocky promotion criteria, as we have two promotions and previous-current-tripleo exists | 15:15 |
| ykarel | ssbarnea, ^^ | 15:15 |
| rlandy | pretty much | 15:19 |
| rlandy | ugh - we have an error | 15:20 |
| rlandy | The error was: 'subnodes' is undefined | 15:20 |
| dtrainor | i'm going to stuff a bunch of things inside of a config/environment/ file, call it with -E, and maybe that will work | 15:24 |
| dtrainor | can i include multiple environment files? | 15:24 |
| *** ykarel has quit IRC | 15:25 | |
| *** chem has quit IRC | 15:26 | |
| rlandy | yes | 15:26 |
| rlandy | panda: review updated | 15:26 |
| *** ykarel has joined #oooq | 15:27 | |
| dtrainor | every time i use this i feel like a new user :) i'll open some doc bugs, maybe they'll help the next person. | 15:28 |
| *** jtomasek has quit IRC | 15:29 | |
| *** ykarel_ has joined #oooq | 15:39 | |
| rfolco | rlandy, I know you are super busy with high priority legacy work.... whats the plan on https://review.openstack.org/#/c/593063? | 15:40 |
| *** kopecmartin has quit IRC | 15:40 | |
| rfolco | I think this needs to merge first --> https://review.openstack.org/#/c/589448/ | 15:41 |
| rfolco | panda, marios rlandy can you have a look at this one please ? we get +2 from sshnaidm already, I also verified as qe, looks sane | 15:42 |
| *** ykarel has quit IRC | 15:42 | |
| rfolco | coffee | 15:45 |
| *** jfrancoa has quit IRC | 15:46 | |
| rlandy | looking | 15:48 |
| rlandy | rfolco: wrt https://review.openstack.org/#/c/593063 | 15:49 |
| rlandy | idk - nothing moves without the reparent reconfigure going through | 15:49 |
| rlandy | it's blocking everything right now | 15:50 |
| rlandy | feel free to move that patch forward | 15:50 |
| rlandy | panda: we will have a merge issue if https://review.openstack.org/#/c/589448/ goes in first | 15:51 |
| rlandy | juts checking multinode failure if related | 15:51 |
| rfolco | rlandy, I thought https://review.openstack.org/#/c/589448/ needs to merge first | 15:52 |
| rfolco | will take a closer look at those | 15:53 |
| rfolco | rlandy, do you mind if I take over if needed today ? | 15:53 |
| rlandy | dtrainor: the grand plan id for us to consolidate/simplify all the CI workflow and reproducers to work through one method - and make that clearly supported. sorry you are caught in our transition time | 15:54 |
| rlandy | rfolco: sure - take over what you wish to to move on - I am watching 596422/ and allies | 15:55 |
| rlandy | still checking your review though - sec | 15:55 |
| dtrainor | all good, i understand it's transition time and what that all might entail. there's a great wealth of information in our docs about what you can do and what's available, but very little in the way of how to actually do it | 15:56 |
| ssbarnea | arxcruz: panda rlandy : i removed linting from https://review.openstack.org/#/c/597547/ -- you should be able to review it, so zuul will merge before tomorrow. | 16:10 |
| rlandy | rfolco: https://review.openstack.org/#/c/589448 - left comments | 16:10 |
| rlandy | panda: ^^ note if this merges - impact on our changes | 16:11 |
| rlandy | ssbarnea: voted on 597547 | 16:12 |
| *** ykarel_ is now known as ykarel|away | 16:16 | |
| *** udesale has quit IRC | 16:16 | |
| arxcruz | ssbarnea: it was my opnion, i'm not core, and i +1 :) | 16:16 |
| *** dsneddon has quit IRC | 16:18 | |
| *** dsneddon has joined #oooq | 16:23 | |
| ssbarnea | arxcruz: no problem, just pointed why I attempted a single commit. Now we have two, and https://review.openstack.org/597581 depends on the fix. I know that most people see linting as futile, but this was a clear case where it would have saved a lot of time. i don't expect many people to check for duplicate keys in a file with >50 lines of code :) | 16:27 |
| *** dsneddon has quit IRC | 16:27 | |
| ykarel|away | ssbarnea, i remember u mentioned timeout in collect logs, seen that here too: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-rocky/19a1653/job-output.txt.gz#_2018-08-29_15_54_25_671407 | 16:32 |
| *** trown is now known as trown|lunch | 16:33 | |
| ssbarnea | ykarel|away: yep, the same as mine, killed after exactly 30m when timeout is configured to 40m....but what seems to be missing is that looking up on the console I see https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-rocky/19a1653/job-output.txt.gz#_2018-08-29_15_52_59_061426 | 16:39 |
| ykarel|away | hmm | 16:41 |
| * ykarel|away leaving | 16:42 | |
| ssbarnea | can someone explain me the ^^ line, and how the REMAINING_TIME really works, i find the concept bit weid. | 16:42 |
| ssbarnea | i don't really see why we should kill a task faster if the previous ones were slower, feels counter intuitive and gives the impression that is somethign wrong with that task. I would rather fast-fail the build instead of starting something that is likely not ever going to suceed. | 16:44 |
| *** ykarel|away has quit IRC | 16:46 | |
| rascasoft | ok rlandy so, ocata passed -> https://review.openstack.org/#/c/597432/ which means that with this and the three in codeng: https://code.engineering.redhat.com/gerrit/#/c/135007/ https://code.engineering.redhat.com/gerrit/#/c/148025/ https://code.engineering.redhat.com/gerrit/148417 I'll finally have master GREEN again :) | 16:51 |
| rascasoft | sshnaidm, if you can have a look also on https://review.openstack.org/#/c/597432/ this will fix all the bm deployments | 16:52 |
| sshnaidm | rascasoft, sure, but we need to see OVB jobs firstly, they are only jobs that uses bm and introspections | 16:53 |
| dtrainor | okie dokie, i'm still getting a deployment failure on an ntpdate update. I've dropped my info here http://paste.openstack.org/show/729066/ | 16:53 |
| rascasoft | sshnaidm, how can I verify that sorry? Aren't the gates enough? | 16:53 |
| *** dtantsur is now known as dtantsur|afk | 16:54 | |
| dtrainor | I used known working examples to create my onfig/environments/default_libvirt_custom.yml but I must be missing something. | 16:54 |
| rascasoft | sshnaidm, because this is quite urgent for rdophase2 | 16:54 |
| sshnaidm | rascasoft, it's not posted yet, I suppose they still run | 16:54 |
| rascasoft | rlandy, what do you think ^^ ? | 16:54 |
| *** ccamacho has quit IRC | 16:54 | |
| sshnaidm | rascasoft, oh, just posted results | 16:56 |
| rascasoft | sshnaidm, and ovb ocata failed... I HATE THIS JOB! | 16:57 |
| sshnaidm | rascasoft, it's failed before that, on undercloud.. because some network issue | 16:57 |
| rascasoft | sshnaidm, yes, I saw it | 16:58 |
| rascasoft | so we just start the recheck dance? | 16:58 |
| sshnaidm | rascasoft, isn't ocata EOLed? | 16:58 |
| rascasoft | sshnaidm, can't say for sure | 16:58 |
| sshnaidm | rascasoft, I'm fine with it, just reran ovb jobs, but not blocking from my side | 17:00 |
| rascasoft | sshnaidm, ok, saw it, thanks | 17:00 |
| rascasoft | rlandy, if I can have your support on the internal three reviews (see above) I can close this: it's been too long that this job has been red :) | 17:01 |
| *** zul has quit IRC | 17:06 | |
| rlandy | yes - looking | 17:11 |
| rlandy | rascasoft: ocata failed on installing the undercloud - so we have no proof - but I'll let it go | 17:13 |
| *** gkadam has joined #oooq | 17:14 | |
| rlandy | rascasoft: I +2'ed but we need to be alert re: ocata | 17:14 |
| rlandy | if we hit an incompatibility | 17:14 |
| rlandy | I won't hold your review on it though | 17:14 |
| *** gkadam has quit IRC | 17:15 | |
| rlandy | rascasoft: merged https://code.engineering.redhat.com/gerrit/#/c/135007/ | 17:15 |
| *** gkadam has joined #oooq | 17:15 | |
| rlandy | https://code.engineering.redhat.com/gerrit/#/c/148025/ merged | 17:16 |
| rascasoft | rlandy++++ | 17:16 |
| rlandy | rascasoft: wrt https://code.engineering.redhat.com/gerrit/#/c/148417/ - I think you would be a lot better off using standard templates and modifying them than keep all your own | 17:17 |
| rlandy | but this is your support burden and I'll accept it if you will | 17:18 |
| rascasoft | rlandy, yes, but the problem here is that I need a customization of the nic_configs, especially for controller and compute | 17:18 |
| rlandy | rascasoft: ok - all merged | 17:19 |
| rascasoft | rlandy, I'll try to optimize it in the future, but for now that's the way it is | 17:19 |
| rascasoft | rlandy, thanks a ton! | 17:19 |
| rlandy | just my $.02 | 17:19 |
| *** vinaykns has quit IRC | 17:21 | |
| rlandy | ssbarnea: seeing this error in other places: http://logs.openstack.org/22/596422/31/check/tripleo-ci-centos-7-scenario009-multinode-oooq/30f6375/job-output.txt.gz#_2018-08-29_16_50_03_604706? | 17:22 |
| ssbarnea | rlandy: yes, in LOTs of places but is likely a reddish herring,, thus is why I created https://review.openstack.org/#/c/597523/ | 17:24 |
| rlandy | ssbarnea: ok - at least I know it's not my patch | 17:25 |
| ssbarnea | rlandy: that is one reason why I am keep raising CRs for eradicating false errors, making output easier to read. any review support would really be appreciated ; | 17:26 |
| ssbarnea | sshnaidm: can you please revise your review on https://review.openstack.org/#/c/588588/ ? -- let me know if you need more explanations. | 17:27 |
| *** dsneddon has joined #oooq | 17:31 | |
| sshnaidm | ssbarnea, does it check something? trying to see in logs http://logs.openstack.org/88/588588/5/check/openstack-tox-linters/0f2c625/job-output.txt.gz but I don't see files output | 17:37 |
| ssbarnea | sshnaidm: yes, it did exact the same thing as before, see http://logs.openstack.org/88/588588/5/check/openstack-tox-linters/0f2c625/job-output.txt.gz#_2018-08-21_09_58_17_593524 | 17:41 |
| sshnaidm | ssbarnea, well, I'm not sure it checked something before too.. | 17:42 |
| sshnaidm | ssbarnea, but I think for this you need to run flake8 with verbosity | 17:42 |
| sshnaidm | ssbarnea, by default it doesn't print anything | 17:42 |
| ssbarnea | sshnaidm: it may be a good idea as it makes debugging easier. flake8 is not very verbose even with -v to worry us. | 17:43 |
| ssbarnea | i will enable it | 17:43 |
| *** jtomasek has joined #oooq | 17:45 | |
| sshnaidm | ssbarnea, just tries, it doesn't print filename even with -v | 17:45 |
| ssbarnea | sshnaidm: but you are right , it was not running repo root so it was linting an almost empty directory. fixing it now. | 17:45 |
| ssbarnea | so my refactoring attempt uncovered that it was not working.... | 17:46 |
| sshnaidm | ssbarnea, iirc we needed to run it on emit* scripts only, but I could be wrong.. panda, rlandy do you remember? | 17:46 |
| ssbarnea | sshnaidm: let me fix it and I will ping you | 17:47 |
| sshnaidm | ssbarnea, sure, thanks | 17:47 |
| *** trown|lunch is now known as trown|brb | 17:50 | |
| *** trown|brb is now known as trown | 17:50 | |
| dtrainor | rlandy, any suggestions for why some of my environment configurations are not sticking? http://paste.openstack.org/show/729066/ | 17:55 |
| *** sanjayu_ has quit IRC | 18:00 | |
| rlandy | sshnaidm: yes - that was our decision at the time | 18:05 |
| ssbarnea | sshnaidm: i updated https://review.openstack.org/#/c/588588/ and obviously that now it required few fixes, I hope we would not need to split it. and again thanks for spotting it, missing to run on all files was clearly a hidden issue. | 18:06 |
| rlandy | dtrainor: the setting is undercloud_undercloud_ntp_servers (two underclouds) your paste has undercloud_ntp_servers as the setting | 18:07 |
| sshnaidm | ssbarnea, yeah, that's expected, fine with me | 18:08 |
| rlandy | panda: :( | 18:08 |
| rlandy | Run the TripleO-CI VXLAN networking script failing out | 18:08 |
| rlandy | 2018-08-29 17:13:58 | [ERROR] /opt/stack/new/devstack/functions-common:Detected fatal | 18:10 |
| rlandy | panda: actually - not failing any worse than other jobs | 18:19 |
| rlandy | sshnaidm: so you know the status of the vxlan failures? | 18:19 |
| rlandy | do you | 18:19 |
| *** zul has joined #oooq | 18:20 | |
| *** vinaykns has joined #oooq | 18:34 | |
| dtrainor | I see, I see. Ok. Is the use of the additional undercloud_ to avoid ambiguity with the undercloud.conf parameter that shares a similar name? | 18:43 |
| *** gkadam has quit IRC | 18:56 | |
| rlandy | that is the undercloud.conf parameter | 19:01 |
| rlandy | undercloud_undercloud is that parameter | 19:01 |
| dtrainor | yes. | 19:03 |
| rlandy | ssbarnea: https://review.openstack.org/#/c/597523/ - still has the vxlan failure | 19:04 |
| rlandy | I would +2 if that showed a fix | 19:04 |
| rfolco | rlandy, I worked on https://review.openstack.org/589448 to rebase that on top of your patch https://review.openstack.org/#/c/593063/ so we don't have conflicts | 19:34 |
| rlandy | rfolco: thanks - I am worried about the conflicts here: https://review.openstack.org/#/c/596422/ | 19:35 |
| rlandy | just going through the zuul errors atm | 19:35 |
| rlandy | which I think are unrelated | 19:35 |
| rlandy | but I won't get this through | 19:35 |
| rlandy | gates | 19:36 |
| rlandy | and there is potential fallout | 19:36 |
| rlandy | and rdo zuul is still queued | 19:36 |
| rlandy | that is my bigger concern | 19:36 |
| rlandy | we are not in a good position here | 19:36 |
| rlandy | we can go nowhere w/o this | 19:37 |
| rlandy | what on earth is a POST failure?? | 19:38 |
| rlandy | ugh | 19:38 |
| rfolco | something wrong with the ansible playbook/role perhaps ? | 19:40 |
| rfolco | rlandy, yes, conflicts very much | 19:40 |
| rfolco | many things were already changed in the 2 patches I mentioned | 19:41 |
| rfolco | this is the main conflicting change https://review.openstack.org/#/c/596422/31/playbooks/tripleo-ci/templates/toci_gate_test.sh.j2 | 19:42 |
| dtrainor | rlandy, does this ring a bell related to the rhos-14.yml release work? "Repository name cannot contain a scheme" http://paste.openstack.org/show/729080/ | 19:48 |
| rlandy | and all the environment files | 19:52 |
| rlandy | rfolco: this is a mess :( | 19:52 |
| * rlandy runs aways | 19:52 | |
| rlandy | dtrainor: I wish I knew | 19:53 |
| rlandy | I haven't had a chance to get our rhos-14 gates going | 19:53 |
| rlandy | as you can from the chatter above we're having a hard time with zuul migration | 19:53 |
| rlandy | sucking the life blood out of other activities | 19:53 |
| rlandy | rhos-14 is on my to-do list for one day - I promise | 19:54 |
| rlandy | rfolco: who let weshay|pto go on PTO anyways??? | 19:54 |
| rfolco | rlandy, lol | 19:55 |
| dtrainor | haha. i've been monitoring how busy it's been, which is why I'm trying to keep my questions to a minimum as best I can :) | 19:56 |
| sshnaidm | rlandy, I'm not aware of vxlan failures | 19:57 |
| rlandy | dtrainor: you're fine - also jschlueter is on pto as well | 19:58 |
| rlandy | sshnaidm: in which context? | 19:58 |
| rlandy | we had some that were our fault | 19:58 |
| sshnaidm | rlandy, to your question.. | 19:58 |
| rlandy | oh - ssbarnea answered | 19:59 |
| rlandy | he has seen those errors elsewhere | 19:59 |
| rlandy | which makes it hard to prove we are ok with getting rid of legacy | 19:59 |
| rlandy | https://logs.rdoproject.org/22/596422/31/openstack-check/legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch/adb2a79/logs/undercloud/home/zuul/vxlan_networking.sh.log.txt.gz#_2018-08-29_17_10_12 | 20:00 |
| dtrainor | rlandy, noted, thanks | 20:03 |
| dtrainor | i'm going to switch over to another task for a bit then. thanks again for the help, learned a lot. | 20:03 |
| rlandy | dtrainor: sure - I hope have bandwidth soon to work through rhos-14 | 20:07 |
| dtrainor | if you ever need help testing or anything please let me know, i think that's an effort i can contribute to | 20:13 |
| rlandy | careful what you put in public chat - we could hold you to that :) | 20:14 |
| dtrainor | hah! | 20:15 |
| rlandy | rfolco: ping - we have another problem | 20:29 |
| rlandy | https://review.openstack.org/#/q/topic:replace_legacy_pre+(status:open+OR+status:merged) | 20:30 |
| rlandy | ^^ of we merge those before we will break stuff | 20:30 |
| rlandy | if | 20:30 |
| rlandy | we can't remove legacy and add new together | 20:31 |
| rfolco | rlandy, will check when I return, will get my son at school, brb | 20:48 |
| *** trown is now known as trown|outtypewww | 21:05 | |
| *** vinaykns has quit IRC | 22:08 | |
| *** rlandy is now known as rlandy|bbl | 22:57 | |
| *** tosky has quit IRC | 23:13 | |
| *** holser_ has quit IRC | 23:29 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!