Thursday, 2021-01-21

*** maharg101 has joined #openstack-ansible00:04
*** maharg101 has quit IRC00:08
*** macz_ has quit IRC00:30
*** dave-mccowan has quit IRC01:24
*** jamesdenton has quit IRC01:33
*** jamesden_ has joined #openstack-ansible01:34
*** maharg101 has joined #openstack-ansible02:04
*** maharg101 has quit IRC02:09
*** cp- has quit IRC02:33
*** cp- has joined #openstack-ansible02:38
*** cp- has quit IRC02:43
*** cp- has joined #openstack-ansible02:45
*** maharg101 has joined #openstack-ansible04:05
*** maharg101 has quit IRC04:10
*** macz_ has joined #openstack-ansible04:21
*** macz_ has quit IRC04:25
*** mgariepy has quit IRC04:53
*** mgariepy has joined #openstack-ansible04:53
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-ansible05:33
*** yasemind has joined #openstack-ansible05:40
yasemindhi, good morning, when i run the openstack-ansible os-zun-install.yml command it is given an error http://paste.openstack.org/show/801807/, i checked container and server have PyMSQL library. can you help me?05:49
yasemindi think it didnt create databases in mysql automatically, but why ?05:54
yasemindCreate database for service is failed05:56
*** ahsen has joined #openstack-ansible06:03
*** pto has joined #openstack-ansible06:09
*** pto has joined #openstack-ansible06:10
*** miloa has joined #openstack-ansible06:46
*** gyee has quit IRC06:46
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible stable/train: Change format of rows  https://review.opendev.org/c/openstack/openstack-ansible/+/77165906:57
*** jamesgibo has joined #openstack-ansible07:15
*** klamath_atx has quit IRC07:18
*** klamath_atx has joined #openstack-ansible07:19
*** dotnetted has quit IRC07:19
*** dotnetted has joined #openstack-ansible07:19
*** luksky has joined #openstack-ansible08:03
*** jamesden_ has quit IRC08:05
*** jamesdenton has joined #openstack-ansible08:05
*** maharg101 has joined #openstack-ansible08:06
*** rpittau|afk is now known as rpittau08:11
*** maharg101 has quit IRC08:12
jrosseryasemind: it would be really useful if you could say which branch of openstack-ansible you are deploying08:15
*** tosky has joined #openstack-ansible08:16
noonedeadpunko/08:17
jrossermorning08:20
jrosserlooks like i need to fix openstack-ansible-tests for the new pip in order to land the other changes08:21
jrosserthe linters job runs from the tests repo but takes global-requirement-pins from the integrated repo08:21
*** andrewbonney has joined #openstack-ansible08:24
*** maharg101 has joined #openstack-ansible08:47
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Use setuptools constraint from global-requirements-pins rather than u-c  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/77177008:49
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Update pip/setuptools/wheel to latest version  https://review.opendev.org/c/openstack/openstack-ansible/+/77028408:50
yasemindjrosser we use ussuri version on our openstack system, but we use stable/victoria for zun09:02
*** ahsen has quit IRC09:05
jrosseryasemind: when you take a role from stable/victoria and try to use it on ussuri you will need to fix up anything which we have changed between those releases09:06
jrosserin particular we have changed the way the db setup is done, so that is going to need some variables overriding09:06
jrosserso for ussuri the db setup is expected to happen on the galera host https://github.com/openstack/openstack-ansible-os_zun/blob/stable/ussuri/defaults/main.yml#L10609:08
jrosserbut we changed this in victoria to the utility host https://github.com/openstack/openstack-ansible-os_zun/blob/stable/victoria/defaults/main.yml#L116-L11709:08
jrosserso you will need to override the variables in the os_zun victoria defaults/main.yml to make it do the db setup on the first galera host09:09
jrosserput these overrides in your user_variables.yml09:09
*** ahsen has joined #openstack-ansible09:11
*** yasemind has quit IRC09:31
*** ahsen has quit IRC09:31
openstackgerritMerged openstack/openstack-ansible-haproxy_server stable/ussuri: Python3 supported version of hatop  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/76873909:32
*** maharg102 has joined #openstack-ansible09:34
*** Anthraxs has joined #openstack-ansible09:34
AnthraxsHello, I'm having an odd issue. I'm trying to use SSL certificates on my AIO installation and the haproxy-install.yml playbook fails with The client lacks sufficient authorization error. I'm using train and I'm trying to use "distro" installation for certobt since certbot auto also trows an error that is deprecated09:36
Anthraxsmy domain name does have propper DNS and it does points to the correct IP...09:36
*** tosky has quit IRC09:36
*** tosky_ has joined #openstack-ansible09:36
*** maharg101 has quit IRC09:37
dotnettedjrosser - you were right with "smells like network trouble" heh - I had both (internal|external)_lb_vip_address on the br-mgmt interface and it was using external_lb_vip_address to communicate with xinetd on galera container - This was not whitelisted in only_from - Dropping the external IP from br-mgmt fixed it - Thanks again09:39
*** tosky_ is now known as tosky09:43
AnthraxsThis is the error that I'm receiving if I use the default configuration (certbot-auto)09:44
Anthraxshttps://pastebin.com/dYtHNSG509:44
jrosserAnthraxs: we did a huge overhaul of the letsencrypt support in later releases than train09:46
*** jamesdenton has quit IRC10:00
*** jamesdenton has joined #openstack-ansible10:01
*** Anthraxs has quit IRC10:14
*** partlycloudy has quit IRC10:16
*** yasemind has joined #openstack-ansible10:17
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump ansible-base and OpenStack collections  https://review.opendev.org/c/openstack/openstack-ansible/+/77178010:19
*** partlycloudy has joined #openstack-ansible10:19
*** ahsen has joined #openstack-ansible10:28
noonedeadpunkSo, we finally get 22.0.0!10:33
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Release Victoria  https://review.opendev.org/c/openstack/openstack-ansible/+/77178610:37
*** yasemind has quit IRC10:43
jrosserwe are really so close to dropping the functional tests entirely from openstack-ansible tests10:56
openstackgerritAndrew Bonney proposed openstack/openstack-ansible-os_neutron master: Prevent neutron-l3-agent killing keepalived on restart  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/77179110:56
*** ahsen has quit IRC10:56
jrosserjust os_neutron being the major user of these now10:56
jrosserwould be really really good to drop the functional tests as it's a huge pain to maintain the tests repo10:56
noonedeadpunkand keystone as well10:59
noonedeadpunkwiht uw scenario10:59
noonedeadpunkBut I think we can move it to integrated pretty easily10:59
*** yasemind has joined #openstack-ansible11:04
noonedeadpunkhm wondering why all neutron functional  tests failed...12:12
jrossersomething wierd is happening, look at this too https://review.opendev.org/c/openstack/openstack-ansible-tests/+/77177012:12
jrosseri needed to fix the 'cannot install setuptools' problem for the linters job12:13
jrosserand somehow * is now broken12:13
noonedeadpunkyeah, and on setup with auth failure12:13
noonedeadpunkmaybe smth is broken with keystone...12:14
jrosseryou've seen some neutron jobs doing the same?12:14
noonedeadpunkyeah for https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/77179112:14
noonedeadpunkhttps://ab9b79f9491924810fc6-88bfc8d18e6df19b9cb6dbfc3489a6ab.ssl.cf1.rackcdn.com/771791/1/check/openstack-ansible-ovs-ubuntu-bionic/e5b3e7d/ for example12:14
jrosseri was thinking my pip change to the tests repo was at fault but it can't be that12:14
noonedeadpunknah, it feels more global then that12:15
jrosserwe've just bumped all the SHA on master12:15
jrosseri was looking in the keystone log, nothing obvious12:15
jrosserthough it's still upset about service tokens, as far as i can see they are just warnings though12:16
noonedeadpunkI can imagine upstream project can drop this functionality now...12:16
*** jamesdenton has quit IRC12:18
*** jamesdenton has joined #openstack-ansible12:19
noonedeadpunkhowever intagrated tests pass, so unlikely...12:23
noonedeadpunk*integrated12:23
jrosserso this is either some unintended consequence from the osa-new-pip changed which have already merged, or theres a bug/change come in with the SHA bump12:25
jrossernoonedeadpunk: i am wondering about this https://github.com/openstack/tempest/commit/cd0bbbdad37a31248d479ef78df948da0a1e850e12:34
yasemindthank you for your helping jrosser and noonedeadpunk , i can install zun and it works12:34
jrosseryasemind: thats great news12:34
noonedeadpunkbackport does not though. Feels I missed smth12:35
noonedeadpunkor on U service was deployed differently12:35
*** yasemind has quit IRC12:40
*** yasemind has joined #openstack-ansible12:40
yasemindnoonedeadpunk yeah we use ussuri openstack system, but we use victoria for zun.12:42
*** jamesgibo has quit IRC12:43
jrosseryasemind: did you see the settings for user-role-requirements? https://docs.openstack.org/openstack-ansible/latest/reference/configuration/extending-osa.html#extend-osa-roles12:44
andrewbonneynoonedeadpunk: are you talking about the zun backport? I've got a pending patch but waiting on kuryr backport first12:44
jrosseryou can use that to make your change to the role version of os_zun persist12:44
jrosseryasemind: if you do a upgrade at any point, re-running bootstrap-ansible would undo anything you've done by hand to change the os_zun branch to victoria12:45
*** jamesgibo has joined #openstack-ansible12:45
jrosserUser 621d7ce0c33c4fd3ae93421a8574926f has no access to domain default _validate_domain_scope12:56
yasemindjrosser yes we updated user-role-requirements, we did stable/victoria12:58
*** rh-jelabarre has joined #openstack-ansible13:02
noonedeadpunkandrewbonney: yeah, was talking about https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/77154713:03
andrewbonneyOk. Once https://review.opendev.org/c/openstack/kuryr/+/771596 merges I've got a patch for the openstack-ansible repo13:03
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible stable/ussuri: Ensure kuryr repo is available within CI images  https://review.opendev.org/c/openstack/openstack-ansible/+/77160813:05
noonedeadpunkah, ok, I see!13:05
noonedeadpunkthaks!13:05
*** tosky has quit IRC13:06
*** klamath_atx has quit IRC13:06
*** tosky has joined #openstack-ansible13:06
noonedeadpunkjrosser: that's weird kind of... are we creating extra default in functional tests?13:16
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: [doc] Release Victoria  https://review.opendev.org/c/openstack/openstack-ansible/+/77180913:18
openstackgerritMerged openstack/openstack-ansible stable/victoria: Disable repeatedly failing zun tempest test  https://review.opendev.org/c/openstack/openstack-ansible/+/77154813:20
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: [doc] Release Victoria  https://review.opendev.org/c/openstack/openstack-ansible/+/77180913:22
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible stable/train: [doc] Clenaup heading page  https://review.opendev.org/c/openstack/openstack-ansible/+/77181013:25
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible stable/train: [doc] Cleanup heading page  https://review.opendev.org/c/openstack/openstack-ansible/+/77181013:26
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible stable/ussuri: [doc] Cleanup heading page  https://review.opendev.org/c/openstack/openstack-ansible/+/77181113:29
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible stable/ussuri: Ensure kuryr repo is available within CI images  https://review.opendev.org/c/openstack/openstack-ansible/+/77160813:31
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible stable/ussuri: Ensure kuryr repo is available within CI images  https://review.opendev.org/c/openstack/openstack-ansible/+/77160813:32
jrossernoonedeadpunk: i was thinking something like that yes, some difference between tempest vars we set in functional vs. AIO13:35
mgariepyjamesdenton, are you around ?13:43
openstackgerritAndrew Bonney proposed openstack/openstack-ansible-os_zun master: Add configuration for zun-wsproxy service  https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/76914313:45
*** owalsh has quit IRC13:48
*** owalsh has joined #openstack-ansible14:08
openstackgerritMerged openstack/openstack-ansible master: Limit threads and processes for Senlin in AIO configuration  https://review.opendev.org/c/openstack/openstack-ansible/+/77125614:08
jamesdentonmgariepy hello there14:22
jamesdentoni was gone. but now i am back.14:23
openstackgerritGaudenz Steinlin proposed openstack/openstack-ansible master: Use TCP mode for console if SSL is configured  https://review.opendev.org/c/openstack/openstack-ansible/+/57415314:24
*** macz_ has joined #openstack-ansible14:25
mgariepyjamesdenton, hey how are you doing ?14:27
mgariepydo you have a deployment with ovs firewall?14:27
jamesdentonno, not anymore. we are still using iptables_hybrid14:28
mgariepyok.14:28
jamesdentonthere was an ugly bug that was only recently fixed (maybe)14:29
mgariepywhat was it ?14:29
jamesdentonone sec14:29
jamesdentonhttps://bugs.launchpad.net/neutron/+bug/173206714:29
openstackLaunchpad bug 1732067 in neutron "openvswitch firewall flows cause flooding on integration bridge" [High,In progress] - Assigned to LIU Yulong (dragon889)14:29
*** macz_ has quit IRC14:30
admin0Your system is not supported by certbot-auto anymore  -- this is for ubuntu focal 20.04 ..14:35
admin0getting that on haproxy_ssl_letsencrypt_enable: true14:35
admin014:35
fricklerthis may be related to the keystone/tempest issues openstack/tempest master: Fix project/domain scope in dynamic_creds  https://review.opendev.org/c/openstack/tempest/+/77181714:38
mgariepyi do have some issues with ovsf firewall it's forgetting some flows..14:38
mgariepyhaha14:38
*** dave-mccowan has joined #openstack-ansible14:39
jamesdentonas in, they are dropping out?14:41
jrosserfrickler: ah awesome, thanks for the link to that, it references the tempest commit that i was suspicious about earlier14:41
mgariepywhen ovsdb and ovs-vswitchd are restarted (like when there is an upgrade of pkg) neutron readd all the flows.14:42
mgariepybut not the remote-sec-group one.14:42
mgariepyis there some doc if i want to build the venv for a specific verison/sha ?14:43
mgariepyi vaguely remember there was but cannot find it :/14:43
jamesdentonmaybe updating this? https://github.com/openstack/openstack-ansible/blob/master/ansible-role-requirements.yml14:44
jamesdentonno14:44
*** sshnaidm|ruck is now known as sshnaidm|afk14:49
mgariepyjamesdenton, here is the output of the flows when ovsdb/vswitchd are restarted: http://paste.openstack.org/show/801819/14:51
*** yasemind has quit IRC14:54
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Move neutron pip packages from constraints to requirements  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/77027615:02
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: L3 agent cleanup_on_shutdown  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/77182615:11
*** spatel has joined #openstack-ansible15:11
*** fridtjof[m] has quit IRC15:20
*** ioni has quit IRC15:20
*** csmart has quit IRC15:20
*** manti has quit IRC15:20
*** irclogbot_2 has quit IRC15:21
*** irclogbot_0 has joined #openstack-ansible15:21
noonedeadpunkspatel: are in correct channel?:)15:27
*** rh-jelabarre has quit IRC15:27
spateldamn it :(15:28
spatelsorry15:28
spatelI need to change my IRC client15:28
spatelhard to know in what channel i am sitting in.15:29
noonedeadpunkok, pool id - what option is that?15:29
spatelMy question was how does osa generate pool id?15:29
spatellet me show you15:29
noonedeadpunkaha in designate.conf15:29
spatelyes in designate.conf has pool_id = 794ccc2c-d751-44fe-b57f-8894c9f5c84215:30
spatelyou have to use same pool_id in pools.yaml otherwise designate doesn't understand what pool you are talking abou15:30
noonedeadpunkuh, nasty15:30
spatelvery ugly15:31
*** rh-jelabarre has joined #openstack-ansible15:31
spatelwe can use "designate-manage pool show_config" command to find ID and use it to generate pools.yamal15:31
*** manti has joined #openstack-ansible15:31
spatelnoonedeadpunk: this document saved my life, how to handle pool_id - https://www.runscripts.com/support/guides/tools/openstack/designate-and-external-dns15:32
noonedeadpunkI'm pretty sure I was not doing all of that...15:32
spatelI spent 24 hour to debug and finally found issue is related to pool_id :)15:32
noonedeadpunkbut I used zookeeper for coordination and that's why I might be ok...15:33
spatelmay be newer version has this pool_id condition. i had no issue with older version of designate15:33
noonedeadpunkor might be that, yes15:33
spateli had no issue related pool id in ussuri deployment15:33
noonedeadpunkyeah I was just doing designate-manage pool update15:34
spatelI did that too but didn't help15:34
spateluntil unless you add id in pools.yaml it doesn't work15:35
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Adds handler for copying policy.json to the right place  https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/74414915:36
spatelThis is what it looks now - http://paste.openstack.org/show/801831/15:36
spatelwith id15:36
spateli will try to run some test to find some workaround15:37
noonedeadpunkuh, designate needs some love for sure15:37
jrosserthat is the default pool id isnt it15:38
noonedeadpunkthey have no mention of it in docs https://docs.openstack.org/designate/victoria/admin/pools.html15:38
spatelin roles we have templates/designate.conf.j2:pool_id = {{ designate_pool_uuid }}15:38
noonedeadpunkjrosser: which is hardcoded in defaults/main.yml15:38
jrosserwell, brace yourself.... https://codesearch.opendev.org/?q=794ccc2c-d751-44fe-b57f-8894c9f5c84215:38
noonedeadpunko_O15:39
spatel:)15:39
noonedeadpunkthat is ridiculous15:39
jrosserdefaults/main.yml maybe but its actually config default inside designate15:39
spateltotally15:39
jrosserso whats this, new behaviour in victoria15:40
spateli had no issue in previous version related pool_id but suddenly with victoria i hit that bug and fix was add id in pools.yaml15:41
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Add dessignate pool uuid to secrets  https://review.opendev.org/c/openstack/openstack-ansible/+/77183315:42
noonedeadpunkI'd expect to see that in docs.. But seems I would need to look through commits instead then15:43
spatelall i am seeing this in commit - https://review.opendev.org/c/openstack/designate/+/75124515:45
andrewbonneyBefore I forget, we hit an issue after a ussuri upgrade with designate not consuming nova/neutron notifications15:47
*** b1tsh1ft3r has joined #openstack-ansible15:47
andrewbonneyMight be some default wiring missing in the designate role15:47
* jrosser boggles at https://review.opendev.org/c/openstack/designate/+/75124515:48
noonedeadpunkI can remember smth like that and I can recall some patches15:48
jrosserreally editing the initial db migration.....?15:48
noonedeadpunkwhich is cherry-picked lol15:48
noonedeadpunkand that means that already deployed designate won't ever be upgraded15:49
noonedeadpunk(to this version)15:49
jrosserright, but they will have the default id15:50
jrosserrather than whatever might have been in the config15:50
jrosserbut even so, just looks super super odd adjusting the first migration in the series15:50
jrossernot my area of expertise so maybe its ok15:51
noonedeadpunkwell looking at changes I think they should have preserved behaviour that was suggested from beginning aren't they?15:51
noonedeadpunkthey're just initializing config before doing migration15:51
noonedeadpunkI faced the same at masakari not so far ago15:52
noonedeadpunkwhen despite what have been set in config - it was ignored because at that step config was not read15:52
noonedeadpunkso default value was used15:52
spatelso this change sitting in master but not in victoria right?15:53
spateldo you think i should also cherry pick for workaround ?15:54
noonedeadpunkand we're doing wrong thing15:55
spatelyes hardcoded uuid15:55
*** sshnaidm|afk is now known as sshnaidm|ruck15:56
b1tsh1ft3rSo, i'm having some issues it seems with a recent upgrade from stein to train 20.2.115:56
b1tsh1ft3rIt would appear that originally, user_variables.yml had neutron_plugin_base defined15:56
b1tsh1ft3rwith only "qos" listed. I believe l3 and metering were a default included thing previously.15:56
b1tsh1ft3rAfter the ugprade i've had problems with l3 agent flapping. I've gone back and since added15:56
b1tsh1ft3rmetering, firewall, l3-agent to the neutron_plugin_base and re-run the plays, however it would15:56
b1tsh1ft3rseem that running 'openstack network agent list' now is showing dead l3 agents.15:56
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Generate designate_pool_uuid dynamically  https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/77184115:56
noonedeadpunkspatel: can you chack ^15:56
b1tsh1ft3rI'm also not able to list routers in the horizon dashboard, or modify quotas of projects without generic errors.15:56
b1tsh1ft3rIt would appear that perhaps maybe rabbitmq traffic is lost from l3-agent to neutron server if that15:56
b1tsh1ft3reven makes sense?! I'm not really seeing anything in the logs either. Running python 2.7 still across15:56
b1tsh1ft3rthe entire stack, but would like to upgrade.15:56
spatelnoonedeadpunk: that looks good, do you want me to run on my lab?15:57
noonedeadpunkyeah, would be awesome, just ensure putting uuid into secrets and generating a value for it15:58
noonedeadpunkb1tsh1ft3r: hey! Was you are the one who reported  https://bugs.launchpad.net/openstack-ansible/+bug/1911482 ?15:58
openstackLaunchpad bug 1911482 in openstack-ansible "neutron-l3-agent broken after train upgrade" [Undecided,New]15:58
spatelyes.. let me try15:58
*** macz_ has joined #openstack-ansible15:59
*** ioni has joined #openstack-ansible15:59
*** fridtjof[m] has joined #openstack-ansible15:59
*** masterpe has joined #openstack-ansible15:59
b1tsh1ft3rnoonedeadpunk co-worker of mine had filed it, but yes same problem. You were dead on about the inventory issue.15:59
*** csmart has joined #openstack-ansible15:59
*** macz_ has quit IRC15:59
*** macz_ has joined #openstack-ansible16:00
noonedeadpunkhm, l3-agent is probably wrong naming for the plugin? I think it's pretty much the list of supported options https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/defaults/main.yml#L328-L33716:02
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Generate designate_pool_uuid dynamically  https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/77184116:07
noonedeadpunkmoreover fwaas is kind of obsoleted so you should consider your options if were using it16:08
b1tsh1ft3rnoonedeadpunk. Only using the following listed under neutron_plugin_base: router, metering, firewall, qos16:08
noonedeadpunkaha, ok16:09
fricklerdesignate recentish changed a bug that essentially prevented starting with a different pool id, likely this is uncovering some other issues here now https://review.opendev.org/c/openstack/designate/+/75124516:09
spatelnoonedeadpunk: do i need to add "id" in this section ? http://paste.openstack.org/show/801787/16:10
noonedeadpunkdo you see anything useful in journalctl -u nautron-l3-agent ?16:10
frickleralso we should probably get rid of that default id being hardcoded16:10
noonedeadpunkI think you don't16:10
spatelcool16:10
spatelrunning playbook now16:10
noonedeadpunkfrickler: yeah that's excatly why https://review.opendev.org/c/openstack/openstack-ansible/+/771833 was pushed at the first place16:11
b1tsh1ft3rnoonedeadpunk http://paste.openstack.org/show/JmzHwDhX8OcRQ2bqJqD4/16:12
b1tsh1ft3rneutron-server however is up16:12
jrosserb1tsh1ft3r: have you checked the health of the rabbitmq cluster?16:15
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Add dessignate pool uuid to secrets  https://review.opendev.org/c/openstack/openstack-ansible/+/77183316:17
spatelnoonedeadpunk: got this DB sync error, let me debug - http://paste.openstack.org/show/801836/16:17
*** jamesgibo has quit IRC16:17
spatelI did drop container and database to re-build designate.16:19
spatelI did run /openstack/venvs/designate-22.0.0.0b2.dev56/bin/designate-manage database sync  on container didn't throw error but echo $? showing 116:22
spatelit did created database in mysql but not sure why sync doesn't like it16:23
noonedeadpunkandrewbonney: I guess that was the fix for designate notifications https://opendev.org/openstack/openstack-ansible-os_nova/commit/881620bd6476d008b218721b70807deca451b8c616:23
noonedeadpunkuh16:23
andrewbonneyYeah, similar for neutron16:23
*** jamesgibo has joined #openstack-ansible16:23
*** jamesgibo has quit IRC16:24
noonedeadpunkspatel: and you have default_pool_id set in config?16:24
spatelyes16:24
spateli used your patch16:24
spateluser_secret.yaml i added uuid16:25
spateland made changes to default/main.yaml and template16:25
noonedeadpunkI'm wondering if it just doesn't like it... what if you set it to 794ccc2c-d751-44fe-b57f-8894c9f5c842 ?:))))16:25
spatelnoonedeadpunk: this is the error i am getting in logs - http://paste.openstack.org/show/801837/16:26
fricklernoonedeadpunk: how do you identify the current pool id? that isn't clear to me from the upgrade reno16:27
noonedeadpunkI think we're just defining random default one?16:27
noonedeadpunknot sure, I made smth stupid?16:27
spatelUUID is just uniq ID and it could be anything to make pool uniq16:28
spatellet me revert my patch and test hold on16:28
b1tsh1ft3rjrosser Yes, rabbitmq is up and running and functioning as expected. checking cluster_status returns all ok and up16:30
mgariepyfun !! https://bugs.launchpad.net/neutron/+bug/191265116:31
openstackLaunchpad bug 1912651 in neutron "ovs flows are not readded when ovsdb-server/ovs-vswitchd are restarted." [Undecided,New]16:31
noonedeadpunkfrickler: ah, I got your qestion. Well, it was always hardcoded to the provided one. So if user has overriden it - then he does not need any futher actions or has it defined somewhere, so it should not be an issue16:31
spatelwhy don't we let user define that UUID so it will be same no matter16:32
noonedeadpunkonce it's generated it will be the same?16:32
*** gyee has joined #openstack-ansible16:32
spatelthat is true16:32
noonedeadpunkwe do that way with literally all uuids and passwords16:32
spatelnoonedeadpunk: i got same error after reverting patch16:33
noonedeadpunkits' just that during upgrade it would be generated in case it's not defined in secrets.yml16:33
spatellook like i need some clean up and re-deploy16:33
noonedeadpunkyou can try dropping tables from designate database16:33
noonedeadpunkor just drop database designate; create database designate;16:33
spatellet me try that16:34
noonedeadpunkand re-run migration16:34
*** jamesgibo has joined #openstack-ansible16:34
fricklernoonedeadpunk: I commented on your patch in the meantime to make it clearer, but I guess that's o.k., then16:35
noonedeadpunkwell, we probably can write some upgrade hook16:37
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Add designate pool uuid to secrets  https://review.opendev.org/c/openstack/openstack-ansible/+/77183316:40
*** jamesgibo has quit IRC16:40
openstackgerritMerged openstack/openstack-ansible-os_manila master: Use global service variables  https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/76994416:45
jrossernoonedeadpunk: i'm not really sure how we can work around the functional tests being broken16:50
jrossertempest is broken for us, so one thing i can see is we could pin this https://github.com/openstack/openstack-ansible-os_tempest/blob/master/defaults/main.yml#L61 temporarily until tempest master is working again16:51
jrosserin the functional tests our SHA bumps do not apply, which is why they are broken but the integrated repo tests are still OK16:51
noonedeadpunkhow great that you made workaround for tempest for the new resolver...16:52
jrosserwell hmm, i guess its not totally helpful for this though16:52
noonedeadpunklet's then set  tempest_git_install_branch to 26.0.0 in test-vars.yml16:53
noonedeadpunkoh, wait...16:54
noonedeadpunkI think it was bumped in u-c?16:54
noonedeadpunkAre we using them?16:54
jrosseros_tempest installs master branch from defaults/main.yml16:55
jrosserlike tip of master, in a functional test16:55
noonedeadpunkwhich won't be the case with new resolver?16:56
noonedeadpunkor...16:56
noonedeadpunkuh...16:56
jrosseryeah16:56
jrosserthe resolver isnt an issue16:56
jrosserit's broken auth behaviour in tempest either way16:56
jrossergood idea with test-vars,i'll try that16:56
noonedeadpunkso, what I was thinking about - if we have it in u-c what not to rely on u-c what tempest version to install? I guess we use tempest version from u-c for stable releases?16:57
jrosserwe do yes16:57
noonedeadpunkwhy we don't use the same loginc for functional tests theb16:57
noonedeadpunk*then16:57
noonedeadpunkjust grab the version that is present in u-c reither installing from source-source16:58
noonedeadpunkor I'm just confused...16:58
jrosserwe'd have to comment this out https://github.com/openstack/openstack-ansible-os_tempest/blob/master/defaults/main.yml#L58-L6116:58
spatelnoonedeadpunk: Your patch failing everytime on db sync (if i revert patch it pass )16:58
noonedeadpunkwe're having so awful weather change today... In the morning it was -22C, now it's -5, in 2 hours it's going to be -2 and tomorrow in the morning +3 or +4. So it's really hard to think for me today :(16:59
spatellet me do small experiment16:59
noonedeadpunkspatel: well....16:59
noonedeadpunkat least we know that16:59
spateli wonder if we need UUID in - - - - formate  like (794ccc2c-d751-44fe-b57f-8894c9f5c842)17:00
spateli used random string with your patch17:00
noonedeadpunkjrosser: ha!@17:00
noonedeadpunkjust try dropping this out https://opendev.org/openstack/openstack-ansible-tests/src/branch/master/test-vars.yml#L36017:01
spatellet me try in - - formate17:01
jrossernoonedeadpunk: oh awesome, good spot17:01
noonedeadpunkand if you define literally 794ccc2c-d751-44fe-b57f-8894c9f5c842 in secrets.yml does migration work?17:01
spateltrying test17:02
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Use setuptools constraint from global-requirements-pins rather than u-c  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/77177017:05
spatelnoonedeadpunk: ready?  794ccc2c-d751-44fe-b57f-8894c9f5c842 in user_secrets.yml PASS playbook17:06
noonedeadpunkgod damn it17:07
spatellet me change some digits and see if it pass again make sure its not hardcoded somewhere17:07
*** b1tsh1ft3r has quit IRC17:09
*** b1tsh1ft3r has joined #openstack-ansible17:10
spatelI changed last 2 digit of UUID and it works not let me change UUID format to reduce some (-) and see17:13
spatels/not/now17:13
noonedeadpunkhm, and what has scripts/pw-token-gen.py generated for you?17:14
spateldamn it i changed to 794ccc2c-d751-44fe-b57f-8888  and it Failed17:15
spatellook like this string has to be in fix numbers and dashes17:15
noonedeadpunkI think it should be just in uuid format17:15
noonedeadpunklike any uuidgen17:15
spatelnoonedeadpunk: i didn't used  scripts/pw-token-gen.py, manually put string in user_secrets.yml17:16
noonedeadpunkand I guess we should generate it as well, but now I'm not 100% sure17:16
noonedeadpunkaha, I see17:16
spatelYes we need to use uuidgen :)17:16
spatellet me generate uuidgen and try that out17:16
noonedeadpunkI think it should be using it https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/pw-token-gen.py#L179-L18117:17
spatellet me try scripts/pw-token-gen.py to see generate secret17:18
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Add designate pool uuid to secrets  https://review.opendev.org/c/openstack/openstack-ansible/+/77183317:19
noonedeadpunkfrickler: tried to update reno and kind of upgrade hook ^17:19
noonedeadpunkworth testing it though....17:19
spatelnoonedeadpunk: ./scripts/pw-token-gen.py works17:23
noonedeadpunkok, nice17:23
spatellook like just need little documentation around so people understand its not password but uuidgen17:24
noonedeadpunkwe rely on secret ending actually17:24
noonedeadpunkwhich we are mising generally...17:24
noonedeadpunkbtw token, key and password are also different17:25
noonedeadpunkit's generally as much as this description https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/pw-token-gen.py#L42-L4517:25
spatelnoonedeadpunk: everything works fine, able to create zone/record etc..17:26
noonedeadpunkecept missing uuid :(17:26
spatelah17:27
spatelwe don't need this patch right because you already included it another patch - https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/77184117:29
spatelsorry may be i am wrong. we do need that17:29
*** dave-mccowan has quit IRC17:31
spatelnoonedeadpunk: how do i apply this patch in production, what is the best way to handle this in prod so next time don't create issue during upgrade17:38
noonedeadpunkyou needto keep designate_pool_uuid in secrets.yml17:41
noonedeadpunkI guess it's the main point. also right now you might want to apply override in designate to have default_pool_id which is commented out now17:42
*** rpittau is now known as rpittau|afk17:42
noonedeadpunkactually I think we need to backport 771841 except pasrt of removing of the uuis17:42
noonedeadpunk*uuid17:42
spateli need to remove hardcoded UUID from roles by hand17:42
noonedeadpunknah, it would just be overriden17:43
spatelwhat about pool_id which you have remove ?17:43
spatelor it doesn't matter17:44
spatellet me run test17:44
spatelhow does this file override when it has hardcoded id  - /etc/ansible/roles/os_designate/templates/designate.conf.j217:45
spateldefault_pool_id = '794ccc2c-d751-44fe-b57f-8894c9f5c842'17:46
*** poopcat has quit IRC17:48
*** poopcat has joined #openstack-ansible17:50
openstackgerritEbbex proposed openstack/ansible-role-systemd_service master: Make systemd.service more consistent  https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/77186917:52
*** miloa has quit IRC17:55
*** cp- has quit IRC17:59
*** maharg102 has quit IRC18:04
*** cp- has joined #openstack-ansible18:08
openstackgerritMarc GariĆ©py proposed openstack/openstack-ansible master: Fix lib/modules path for focal release  https://review.opendev.org/c/openstack/openstack-ansible/+/77187018:12
*** b1tsh1ft3r has quit IRC18:13
*** cp- has quit IRC18:14
admin0anyone has centralized logging ?18:19
admin0what is being used18:19
admin0also what do you guys use for monitoring18:20
*** jamesgibo has joined #openstack-ansible18:21
*** jamesgibo has quit IRC18:23
*** jamesgibo has joined #openstack-ansible18:24
noonedeadpunkgraylog / zabbix?:)18:27
*** cp- has joined #openstack-ansible18:30
*** poopcat has quit IRC18:31
*** poopcat has joined #openstack-ansible18:32
*** jamesgibo has quit IRC18:32
jrosseroh no not another one https://github.com/openstack/requirements/blob/378d1f4adfbf8c067b8a802075b5805156170057/upper-constraints.txt#L12218:36
jrosser"horizon works as a library for horizon plugins" <- thats unfortunate18:38
*** jamesdenton has quit IRC18:43
*** jamesdenton has joined #openstack-ansible18:43
spateladmin0: graylog18:46
spatelwhat do you guys using to ship journactl logs to graylog?18:47
spatelI am planning to use syslog hook which is easy to implement18:47
*** strattao has joined #openstack-ansible18:49
*** andrewbonney has quit IRC18:55
admin0i have graylog, but the journale2gelf dies ( when used from ops git)18:56
admin0so checking if anyone has a successful "recent" implementation18:57
admin0how is openstack lb set -status error  dangerous ?19:01
admin0just wondering19:01
admin0people would immediately delete it after that19:02
admin0no one wants to see errors in their list19:02
admin0sorry .. wrong channel19:06
admin0:)19:06
kleinispatel: http://paste.openstack.org/show/801846/ <- this is my very dynamic version of pools.yaml definition. I have PowerDNS running on infra hosts. Maybe this can be helpful19:07
spatelkleini: nice!!19:09
*** pto_ has joined #openstack-ansible19:10
kleiniBut be careful, I have issues with that. I don't get all PowerDNS instances on the infra hosts to fetch latest zones. But this is maybe caused for my deployment by adding an infra host after PowerDNS and some zones are deployed. The IN SOA records do not contain all PowerDNS instances.19:13
*** pto has quit IRC19:13
spatelI have external PowerDNS19:13
kleiniGenerally I configured every designate-mdns to be master for every PowerDNS instance and every PowerDNS instance is one target. So I don't know, if this setup is correct to have all cross connected.19:14
spatelIn my case i am planning to use mdns to be master and my external powerDNS will be slave19:15
kleiniI did the same19:19
kleiniBut I still get new zones into only one PowerDNS instance and not in all. That is my problem.19:19
spatelhmm i will try and let you know if hit same issue19:26
spatelcurrently i have single powerDNS19:26
*** MrClayPole has quit IRC19:27
*** gixx has quit IRC19:28
*** MrClayPole has joined #openstack-ansible19:28
*** gixx has joined #openstack-ansible19:28
dotnettedDuring ceph-install.yml, a ceph volume is created on the target host but task [ceph-osd : wait for all osd to be up] runs inside infra1_ceph-mon_container which has no knowledge of this volume. This causes the ceph osd stat in the container to fail. What might I be doing wrong here? Thanks19:32
dotnettedIs the idea for the container ceph to communicate w/ ceph on the host or should the device be mounted into the container for local access?19:33
*** dave-mccowan has joined #openstack-ansible19:54
admin0dotnetted, you can also have the ceph up and then just use the conf to have osa talk to ceph19:55
admin0this way, you can work with ceph-ansible independently of osa19:56
dotnettedis that the more common use case?19:56
dotnettedlooks like "brosky" asked the same thing back in april but left before figuring it out heh: http://104.130.124.113/irclogs/%23openstack-ansible/%23openstack-ansible.2020-04-24.log.html19:57
dotnettedExcept for the fact that I have "devices:" defined and the ceph-volume is being properly created on the target host19:58
*** spatel has quit IRC20:00
*** jamesdenton has quit IRC20:06
*** jamesdenton has joined #openstack-ansible20:06
*** irclogbot_0 has quit IRC20:14
*** tosky has quit IRC20:14
admin0dotnetted, that would be a recommended use case .. this way, you will be able to use the best of both20:14
dotnettedthanks for the info :)20:15
*** tosky has joined #openstack-ansible20:15
*** irclogbot_2 has joined #openstack-ansible20:16
mgariepyrestart mysql(all) ?20:41
mgariepy:/20:41
*** spatel has joined #openstack-ansible20:43
mgariepyhttps://github.com/openstack/openstack-ansible-galera_server/blob/master/handlers/main.yml#L73-L9020:44
spatelkleini: hey do you have multiple Slave DNS in options: ?20:55
spatelI have more than 2 slave PowerDNS then i need to notify both of them right and only option to do that is define multiple DNS in options:20:56
spatelah! i found also_notifies:20:58
spatelthat is what we need20:58
jrossermgariepy: https://github.com/openstack/openstack-ansible/blob/master/playbooks/galera-install.yml#L44 ?21:13
*** owalsh has quit IRC21:26
*** dotnetted has quit IRC21:44
*** owalsh has joined #openstack-ansible21:44
*** owalsh has quit IRC22:16
admin0I  have this command: neutron subnet-create --allocation-pool start=xxxx:yyyy:8011:0000:0000:0000:0000:1111,end=xxxx:yyyy:8011:ffff:ffff:ffff:ffff:ffff  --ip-version 6  --no-gateway --host-route destination=::/0,nexthop=xxxx:yyyy:8011::1 --enable-dhcp --name subnet6-direct --dns-nameserver 2001:4860:4860::8888  UUID  xxxx:yyyy:8011::/48  --- .. when i login to the vm, the gateway is not set to xxxx:yyyy:8011::1 but something else ( not even in22:20
admin0this range) .. what am I doing wrong ?22:20
spatelkleini: i figured out how to send notification to multiple slave powerDNS22:21
spateladmin0: your command looks good so that is strange if it set gateway something else. make sure you don't have any other DHCP running on same VLAN or SLAAC may be causing this issue22:24
*** owalsh has joined #openstack-ansible22:25
admin0hmm..  did not thought of another dhcp server on that vlan ..22:27
admin0will check22:28
*** spatel has quit IRC22:37
*** d34dh0r53 has quit IRC22:37
*** d34dh0r53 has joined #openstack-ansible22:48
*** maharg101 has joined #openstack-ansible23:26
*** maharg101 has quit IRC23:31
*** strattao has quit IRC23:38
*** luksky has quit IRC23:51
*** lemko has quit IRC23:51
*** lemko6 has joined #openstack-ansible23:51

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!