Thursday, 2020-07-16

*** Adri2000 has quit IRC00:41
*** Adri2000 has joined #openstack-ansible00:48
*** cshen has joined #openstack-ansible00:53
*** cshen has quit IRC00:58
*** cshen has joined #openstack-ansible02:54
*** cshen has quit IRC02:59
*** dave-mccowan has quit IRC03:30
*** rh-jelabarre has quit IRC03:46
*** gyee has quit IRC04:15
*** cshen has joined #openstack-ansible04:55
*** udesale has joined #openstack-ansible04:57
jrosserarkan: the heat stack domain admin role should be configured here https://github.com/openstack/openstack-ansible-os_heat/blob/master/tasks/heat_service_setup.yml#L67-L7105:17
jrosserarkan: and for this bug https://bugs.launchpad.net/openstack-ansible/+bug/1877421 the comments are a bit confusing, one saying train/nautilus is OK and another saying it is not?05:21
openstackLaunchpad bug 1877421 in openstack-ansible "Cinder-volume is not able to recognize a ceph cluster on OpenStack Train." [Undecided,New]05:21
*** cshen has quit IRC06:03
*** this10nly has joined #openstack-ansible06:36
*** csmart has joined #openstack-ansible06:38
*** arkan has quit IRC06:50
*** andrewbonney has joined #openstack-ansible07:03
*** shyamb has joined #openstack-ansible07:05
*** arkan_ has joined #openstack-ansible07:12
*** arkan_ is now known as arkan07:12
*** shyam89 has joined #openstack-ansible07:18
*** shyamb has quit IRC07:21
arkanjrosser: I've changed in openstack_user_config.yml under storage_hosts: cinder_backends: ceph: rbd_user:07:22
arkanfrom cinder ---> ceph07:22
arkanthen run  os-cinder-install.yml07:23
arkanthis will change rbd_user = cinder in [ceph] (/etc/cinder/cinder.conf) in the container to rbd_user = ceph07:23
arkannow the issue disappeared, but another one appeared07:24
arkanhttp://paste.openstack.org/show/795968/07:25
arkansomething with rados.PermissionDeniedError07:25
*** udesale has quit IRC07:26
arkanAnd about ceph nautilus, I have it installed separately with ceph-ansible, there is no problem with it07:26
arkanalso from openstack glance-images, and ephemeral-vms pools are working07:27
arkanI have images in ephemeral-vms and glance-images pools and no problem about these07:28
*** shyam89 has quit IRC07:28
arkanI'm using rbd from ceph, but I'm not using ceph fs07:30
*** udesale has joined #openstack-ansible07:31
*** yasemind has joined #openstack-ansible07:43
jrosserarkan: i think that rbd_user should be cinder, not ceph07:46
arkanjrosser: ok, then I will get the same error again if I will set it, I will put it back07:47
jrosserarkan: the thing that is odd in that bug report is this "infra1-cinder-volumes-container-b943c727"07:49
jrosserwe don't normally run a container for cinder-volume service07:50
arkanjrosser: I think rbd_user is ceph not cinder07:50
arkanone moment to verify07:50
arkanroot@controller1:~# ps aux | grep ceph07:51
arkanroot        7227  0.0  0.0  46256 11468 ?        Ss   05:58   0:00 /usr/bin/python2.7 /usr/bin/ceph-crash07:51
arkanceph        8298  1.9  0.4 1452804 325484 ?      Ssl  05:58   2:14 /usr/bin/ceph-mgr -f --cluster ceph --id controller1 --setuser ceph --setgroup ceph07:51
arkanceph        8329  1.2  0.3 710092 254556 ?       Ssl  05:58   1:26 /usr/bin/ceph-mon -f --cluster ceph --id controller1 --setuser ceph --setgroup ceph07:51
arkanceph        8446  0.4  0.3 1076604 242988 ?      Ssl  05:58   0:33 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph07:51
arkanceph        8453  0.4  0.3 1112232 276244 ?      Ssl  05:58   0:30 /usr/bin/ceph-osd -f --cluster ceph --id 3 --setuser ceph --setgroup ceph07:51
arkanoh07:51
arkanpastbin07:51
arkanhttp://paste.openstack.org/show/795970/07:51
arkanusually something like this must be ceph@ceph07:51
arkannot cinder@ceph07:51
jrosserbut isnt that the user that the ceph daemons are running under07:52
jrosserthe unix user07:52
jrosserthats really not the same as the user/keyring for client access07:52
arkanok07:52
arkanone moment07:52
jrosserfor example07:52
jrosserroot@infra1:~# ls -l /etc/ceph/*.keyring07:52
jrosser-rw-r----- 1 root ceph 82 May 14  2019 /etc/ceph/ceph.client.cinder.keyring07:52
arkanceph auth list --> http://paste.openstack.org/show/795971/07:53
arkanI will check also /etc/ceph/*.keyring07:53
arkanin the container07:53
arkanceph.client.cinder.keyring (ciner volume container) ---> http://paste.openstack.org/show/795972/07:55
jrosseryes, so in this case the name of the client for cinder is "cinder"07:55
arkanok07:55
arkanthen I will put it back to cinder07:55
jrosseri have a similar setup here with a seperate ceph cluster made with ceph-ansible so we can compare things if you like07:56
arkanalso there is another settings that I've used in user_variables:07:57
arkancinder_ceph_client: cinder07:57
arkansure07:57
arkanlet's compare07:57
jrosseruser variables for ceph http://paste.openstack.org/show/795973/07:58
arkanok07:58
arkanI will past also mine 2 files07:59
jrosseropenstack user config for storage hosts http://paste.openstack.org/show/795975/07:59
arkanopenstack_user_config.yml ---> http://paste.openstack.org/show/795974/07:59
*** shyamb has joined #openstack-ansible07:59
arkanuser_variables.yml ---> http://paste.openstack.org/show/795976/08:00
*** cshen has joined #openstack-ansible08:00
*** tosky has joined #openstack-ansible08:02
*** cshen has quit IRC08:04
arkanfrom what I can see, it seems ok08:04
jrosseras i understand it 'rbd_user' is whats passed as the client name to the rbd command08:05
jrosserwhich then makes it find the corresponding keyring in /etc/ceph08:05
arkanok08:07
arkanbut this is strange, that I have glance and ephemeral are working08:07
arkanI have images there08:07
arkanand I can upload images08:07
arkanmy ceph health is OK, greeen08:08
arkan*green08:08
arkanmaybe if I solve this problem then magnum will work08:09
arkanbecause it could not create volume08:10
*** shyam89 has joined #openstack-ansible08:13
arkanI put the config back and rerun ansible os cinder08:15
*** shyamb has quit IRC08:16
*** tosky has quit IRC08:16
arkannow it's throwing cinder.exception.ClusterNotFound: Cluster {'name': 'ceph@ceph'} could not be found08:16
arkanstrange08:16
arkanbut from command line it works08:18
arkancli (cinder volume) ---> http://paste.openstack.org/show/795977/08:18
arkanand my config was working sometimes ago08:19
arkanin the old deployments, I'm using the same config for cinder volumes08:20
arkanbut in the recent deployment it's not working08:20
arkanI don't know if something has changed in the meantime, because I'm like CI/CD, I made a lot of deployments08:21
arkan:))08:21
*** stingrayza has joined #openstack-ansible08:23
*** also_stingrayza has quit IRC08:24
jrosserarkan: you also have a cinder-volumes container?08:25
arkanI pasted from inside controller1-cinder-volumes-container-1f5a934808:26
arkanisn't it ?08:26
*** shyam89 has quit IRC08:30
*** tosky has joined #openstack-ansible08:33
jrosserarkan: are you able to paste some of the error you are getting from cinder-volume log?08:35
arkanone moment08:36
arkanjournalctl -xf (from cinder volumes container), debug = False :  http://paste.openstack.org/show/795981/08:38
arkanI can also turn the debug = True, to get more08:38
jrosserright, so i get this INFO cinder.service [req-0c7ef275-78e4-4761-aaf0-9fc2d7e2946a - - - - -] Starting cinder-volume cluster ceph@rbd_hdd (version 15.1.1)08:43
arkanhumm ceph@rbd_hdd !!08:44
jrosserand that is corresponding to volume_backend_name: rbd_hdd from openstack_user_config08:44
arkanI have it volume_backend_name: ceph08:45
jrosserand that in turn then corresponds to a config section in /etc/cinder/cinder.conf http://paste.openstack.org/show/795982/08:46
*** shyamb has joined #openstack-ansible08:47
arkanin my case it's  Starting cinder-volume cluster ceph@ceph (version 15.1.1)08:50
arkanand that is corresponds to the section config in /etc/cinder/cinder.conf08:51
jrosseryes, looks like the names all line up08:51
arkanhttp://paste.openstack.org/show/795983/08:51
arkanI'm sure that I could created volumes in the past deployment08:52
arkan*deployments08:52
jrosserwhat about rbd_secret_uuid?08:53
arkanI did not use it at all in all my deployments08:53
arkanand I could created volumes, volumes were working08:53
*** cshen has joined #openstack-ansible08:54
jrosserarkan: really as always the reference should be the AIO build, thats what we test09:08
arkanjrosser: thanks, I will check further09:09
*** shyamb has quit IRC09:22
*** cshen has quit IRC09:31
jrosserarkan: i have run an AIO here and it does this cinder.service [req-c70b93c4-06e2-41e8-9254-c7d032a153b4 - - - - -] Starting cinder-volume cluster ceph@RBD (version 16.1.0)09:39
jrosserthis is the section in cinder.conf http://paste.openstack.org/show/795985/09:40
jrosseri just wonder if there is some bug in cinder somewhere with the section name being called 'ceph' or its some reserved word somewhere09:41
*** shyamb has joined #openstack-ansible09:45
*** pamsoo has joined #openstack-ansible09:45
*** arkan has quit IRC09:50
*** arkan has joined #openstack-ansible09:50
pamsoohi. i'm wondering that this value(https://github.com/openstack/openstack-ansible/blob/stable/train/inventory/group_vars/all/keystone.yml#L27) might be invalid.09:52
pamsooi think that keystone_service_adminuri_insecure should be 'false' when keystone_service_adminuri_proto is https, but it's opposite. am i correct?09:52
pamsooany reply will helpful to me :-)09:55
*** cshen has joined #openstack-ansible10:07
*** cshen has quit IRC10:12
*** pamsoo has quit IRC10:24
*** sakharkar has joined #openstack-ansible10:28
sakharkarnoonedeadpunk: As OSA U is supporting Ubuntu20, is it under development or we can deploy it as on Ubuntu 1810:29
sakharkarAs OSA U is supporting Ubuntu20, is it under development or we can deploy it as on Ubuntu 18? Doc is not updated for Ubuntu2010:31
*** shyamb has quit IRC10:36
*** cshen has joined #openstack-ansible10:41
jrosserpamsoo is gone but the answer there is that keystone_service_adminuri_insecure should be true when the protocol is https but the certificate is self signed10:44
jrossera self signed certificate is untrusted and required insecure=True10:45
*** cshen has quit IRC10:45
jrossersakharkar: OSA U is still a release candidate and not yet released, but the plan is to support ubuntu 2010:45
jrossersakharkar: if you are going to test in a lab environment it would be good to get feedback on Ussuri/Focal10:46
*** shyamb has joined #openstack-ansible10:55
CeeMacbefore i go check the nova channel, has anyone here any expierence with quem-guest-agent not working?  virsh guest-ping isn't working11:06
CeeMacand if I dump the xml of the instance. the socket config has a state of disconnected11:07
CeeMacbut I' ve no clue what variable in nova would affect that at instance creation. the only relevant image property I've seen was enabled on the image I deployed the test instance from11:07
*** mgariepy has quit IRC11:10
watersji get this error when trying to access masakari webui (train, ubuntu, source install) http://paste.openstack.org/show/795993/11:20
watersjyes, its self-sign cert. and doesn't appear to have a CA in the chain (inhouse).11:23
jrosserwatersj: it should not really be trying to contact the external endpoint in the ui code11:37
jrossermost likley this is a wrong or missing config for masakari, but I don’t run that  here unfortunately11:39
watersjits internal to my network, and ip it resolve to is the "external_lb_vip_address:"11:39
watersjshould be using "internal_lb_vip_address:"11:40
watersjdouble checking service definitions http://paste.openstack.org/show/795994/, its using the public service11:56
*** rh-jelabarre has joined #openstack-ansible11:59
*** shyamb has quit IRC12:13
*** mgariepy has joined #openstack-ansible12:17
*** cshen has joined #openstack-ansible12:42
*** cshen has quit IRC12:46
*** cshen has joined #openstack-ansible12:49
*** cshen has quit IRC12:54
*** andrewbonney has quit IRC12:56
towjrosser: you nailed it, repo_hosts was the issue13:04
*** spatel has joined #openstack-ansible13:04
*** pamsoo has joined #openstack-ansible13:13
jrossertow: cool - is it all looking a bit more sensible now?13:18
pamsoowatersj: thank you for replying. then, are you saying 'insecure' means that verifying certificate is not necessary and doesn't mean that messages will not be encrypted?13:18
pamsoooh, sorry watersj. i mean jrosser. ^^;;;13:18
jrosserinsecure means that the certificate should not be trusted13:19
jrosserit will be encrypted, but the cert is not checked as being valid against the store of CA certificates13:20
jrosserpamsoo: the logic is both of these lines https://github.com/openstack/openstack-ansible/blob/stable/train/inventory/group_vars/all/keystone.yml#L29-L3013:20
pamsooi was misunderstanding. thank you for correcting me.13:20
jrosserno problem13:21
towjrosser: yeah, as far as OSA goes things look good, having some weird yum issues though13:21
*** pamsoo has quit IRC13:24
*** dave-mccowan has joined #openstack-ansible13:25
*** andrewbonney has joined #openstack-ansible13:37
openstackgerritAndrew Bonney proposed openstack/openstack-ansible master: DNM: Test nova changes from OSA repo  https://review.opendev.org/74144913:44
openstackgerritAndrew Bonney proposed openstack/openstack-ansible-os_nova master: Use Nova management IP for live migrations  https://review.opendev.org/74115513:51
openstackgerritAndrew Bonney proposed openstack/openstack-ansible-os_nova master: Use nova_management_address as a default VNC bind address  https://review.opendev.org/74115613:58
openstackgerritAndrew Bonney proposed openstack/openstack-ansible master: DNM: Test nova changes from OSA repo  https://review.opendev.org/74144914:03
*** ianychoi has joined #openstack-ansible14:05
skellyis there any documentation on how to do OS upgrades? I've done a few but am stuck on how to upgrade the first host in the host group14:05
*** this10nly has quit IRC14:08
*** cshen has joined #openstack-ansible14:50
*** cshen has quit IRC14:54
*** yasemind has quit IRC14:55
*** schwicht has quit IRC14:55
*** miouge362536868 has quit IRC14:55
*** irclogbot_2 has quit IRC14:55
*** alvinstarr has quit IRC14:55
*** maharg101 has quit IRC14:55
*** admin0 has quit IRC14:55
*** yasemind has joined #openstack-ansible14:57
*** schwicht has joined #openstack-ansible14:57
*** miouge362536868 has joined #openstack-ansible14:57
*** irclogbot_2 has joined #openstack-ansible14:57
*** alvinstarr has joined #openstack-ansible14:57
*** maharg101 has joined #openstack-ansible14:57
*** admin0 has joined #openstack-ansible14:57
*** mloza has joined #openstack-ansible15:00
*** spatel has quit IRC15:04
*** gyee has joined #openstack-ansible15:25
*** dave-mccowan has quit IRC15:28
*** udesale_ has joined #openstack-ansible15:32
*** udesale has quit IRC15:34
arkanjrosser: "i have run an AIO here and it does this cinder.service [req-c70b93c4-06e2-41e8-9254-c7d032a153b4 - - - - -] Starting cinder-volume cluster ceph@RBD (version 16.1.0)"16:24
arkanThanks, I will make a change for RBD16:24
arkanand see16:24
*** udesale_ has quit IRC16:25
*** sshnaidm is now known as sshnaidm|afk16:28
jrosserarkan: ok, remember that theres nothing special about "RBD" it's just the name given to the backend16:35
arkanyes16:36
arkanjrosser: but you said maybe it's a reserved word "ceph"16:37
jrossermaybe yes, just a hunch16:37
*** mgariepy has quit IRC16:45
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Add zun to required-projects  https://review.opendev.org/74149316:45
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_zun master: Add placement client to zun config file  https://review.opendev.org/74149416:48
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_zun master: Add placement client to zun config file  https://review.opendev.org/74149416:48
*** arkan has quit IRC16:50
*** cshen has joined #openstack-ansible16:50
*** arkan has joined #openstack-ansible16:50
*** cshen has quit IRC16:54
arkanjrosser: I have this installed for ceph http://paste.openstack.org/show/796018/17:13
arkanvolume type = ceph, volume_backend_name = ceph17:13
*** mgariepy has joined #openstack-ansible17:26
*** andrewbonney has quit IRC17:46
*** cshen has joined #openstack-ansible18:50
*** cshen has quit IRC18:54
*** cshen has joined #openstack-ansible19:13
*** nicolasbock has quit IRC19:14
*** nicolasbock has joined #openstack-ansible19:15
*** d34dh0r53 has quit IRC19:20
*** cshen has quit IRC19:23
*** cshen has joined #openstack-ansible19:26
*** d34dh0r53 has joined #openstack-ansible19:28
*** cshen has quit IRC19:30
arkanjrosser: the volumes are working now19:31
arkanthis is odd19:32
arkanI just add rbd which correspond to the section [rbd] in /etc/cinder/cinder.conf19:32
arkaneven 'ceph' type is working19:32
arkanI've deleted 'ceph' type because I don't want it now19:35
arkanopenstack volume type delete ceph19:35
arkanand I can create volumes now19:35
*** sakharkar has quit IRC19:37
arkanmy openstack_user_config.yml now it's using rbd as key for cinder_backends: which corresponds to [rbd] in /etc/cinder/cinder.conf and which corresponds to a volume type19:38
jrosserarkan: so everything is aligned with the key 'rbd'19:44
arkanyes19:44
jrosserwhen you were using 'ceph' did it all look aligned the same, just not working?19:45
arkanyes, it was aligned19:45
jrosserthat is really odd19:45
arkanyes19:45
arkanI don't know, maybe something have to do with the keyword ceph19:45
arkanbut this is odd19:45
jrosserand we see from the AIO that 'RBD' is also OK, so different strings can work19:45
jrosservery strange indeed19:46
arkanand I only re-run os-cinder-install.yml as usual19:46
arkananyway, now I returned to magnum :))19:46
arkanI can see this error magnum.common.exception.GetDiscoveryUrlFailed: Failed to get discovery url from 'https://discovery.etcd.io/new?size=1'.19:47
arkanbut why ?19:47
arkanthe link seems to work19:47
arkanand if I curl from inside the magnum container, it works19:48
arkanalso I caught this strange policy file which I could not find in /etc/magnum/19:49
arkan2020-07-16 19:43:53.555 138 ERROR magnum.drivers.heat.k8s_fedora_template_def [req-8607a2a1-8bf8-481b-a1c6-58c356148c99 - - - - -] Failed to load default keystone auth policy: FileNotFoundError: [Errno 2] No such file or directory: '/etc/magnum/keystone_auth_default_policy.json'19:49
*** arkan has quit IRC19:50
*** arkan has joined #openstack-ansible19:50
arkanjournalctl -xf (magnum container) ---> http://paste.openstack.org/show/796026/19:51
*** cshen has joined #openstack-ansible20:02
*** theintern_ has joined #openstack-ansible20:05
*** vesper11 has quit IRC20:20
*** vesper11 has joined #openstack-ansible20:21
*** schwicht has quit IRC20:35
*** schwicht has joined #openstack-ansible20:36
*** arkan has quit IRC20:50
*** arkan has joined #openstack-ansible20:50
*** schwicht has quit IRC20:50
*** cshen has quit IRC21:03
*** jbadiapa has quit IRC21:20
*** markvoelker has joined #openstack-ansible21:23
*** markvoelker has quit IRC21:26
*** this10nly has joined #openstack-ansible21:28
*** markvoelker has joined #openstack-ansible21:33
*** markvoelker has quit IRC21:38
*** arkan has quit IRC21:50
*** yasemind has quit IRC22:03
*** aedc_ has joined #openstack-ansible22:36
*** aedc has quit IRC22:39
*** aedc_ has quit IRC22:52
*** idlemind has quit IRC22:56
*** aedc has joined #openstack-ansible22:56
*** aedc has quit IRC22:57
*** aedc has joined #openstack-ansible22:58
*** cshen has joined #openstack-ansible23:00
*** tosky has quit IRC23:04
*** theintern_ has quit IRC23:04
*** cshen has quit IRC23:05
*** idlemind has joined #openstack-ansible23:10
*** aedc has quit IRC23:10
*** gyee has quit IRC23:16
*** miouge362536868 has quit IRC23:16
*** irclogbot_2 has quit IRC23:16
*** alvinstarr has quit IRC23:16
*** maharg101 has quit IRC23:16
*** admin0 has quit IRC23:16
*** gyee has joined #openstack-ansible23:23
*** miouge362536868 has joined #openstack-ansible23:23
*** irclogbot_2 has joined #openstack-ansible23:23
*** alvinstarr has joined #openstack-ansible23:23
*** maharg101 has joined #openstack-ansible23:23
*** admin0 has joined #openstack-ansible23:23
*** this10nly has quit IRC23:32

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!