Wednesday, 2021-04-28

*** CeeMac has quit IRC00:12
*** priteau has quit IRC02:03
*** rh-jelabarre has quit IRC02:21
*** evrardjp has quit IRC02:33
*** evrardjp has joined #openstack-ansible02:33
openstackgerritXinxin Shen proposed openstack/openstack-ansible-os_masakari master: setup.cfg: Replace dashes with underscores  https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/78839903:19
*** miloa has joined #openstack-ansible03:37
*** miloa has quit IRC03:59
*** shyamb has joined #openstack-ansible05:32
*** pto has joined #openstack-ansible06:30
*** luksky has joined #openstack-ansible06:34
*** gyee has quit IRC06:35
*** gokhani has joined #openstack-ansible07:11
*** andrewbonney has joined #openstack-ansible07:15
gokhaniHi folks, I deployed OSA (22.1.0) with ceph integration. Ceph cluster and cinder integration works properly, but on swift side I am getting 401 unauthorized errors. radosgw and keystone connection seems broken. I added these rows in user_variables.yml >> http://paste.openstack.org/show/804817/ . this file shows radosgw logs >07:18
gokhanihttp://paste.openstack.org/show/804818/ Dou you have any ideas about this problem? May be I am missing something to do07:18
* recyclehero fianlly sees some movement on the channel07:23
*** shyamb has quit IRC07:24
gokhaniand also it seems radosgw behind haproxy doen't work properly. Some requests are failing > http://paste.openstack.org/show/804819/07:26
*** shyamb has joined #openstack-ansible07:36
*** rpittau|afk is now known as rpittau07:36
noonedeadpunklet me see07:36
noonedeadpunkdid it end up in proper ceph.conf?07:37
noonedeadpunkAlso I think you might be missing `rgw_frontends = civetweb port=$LOCAL_IP:$RGW_PORT`07:39
noonedeadpunkfwiw, these are defaults https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/ceph-rgw.yml so wondering why you decided to override them?07:40
noonedeadpunkCan you share your actual ceph.conf? As I'd suggest that there are extras, since these are only overrides07:45
recycleherohi, my nfs backend was a raid array. there was some problems with the array and I had to use backups and create a new array.07:48
recycleheronow some instances went to grub resuce07:48
recycleheroand the drives owner and group are wierd in the nfs backedn07:49
recyclehero-rw-r--r-- 1 64055 64055 16106127360 Apr 28 02:45 volume-501dd125-f8b8-4e6f-b142-15ada81c68ad07:49
noonedeadpunksorry, no idea how can help here :(07:50
recycleherohow can I enable logging for the lxc containers? I dont see their logs in the lxc host.07:52
recycleherothe folder /var/log/lxc is there but their atime is for way back07:54
*** mathlin has joined #openstack-ansible07:55
recycleheroshould nfs share owner:group and mode be something specefic?07:57
recycleherothey are  libvirt-qemu:libvirt-qemu on the compute host and 64055:64055 on the nfs backend07:59
recycleherois this normal?07:59
noonedeadpunkrecyclehero: yes, last part is normal, since permissions should be meaningfull on hosts at the first place. But with that, all hosts should have same uids and guids then08:05
recycleheronoonedeadpunk: noonedeadpunk: now that the instances are shutoff they are back to root:root 644 on the nfs host08:09
recycleheronoonedeadpunk: do you know how to see  lxc containers logs08:10
recycleheroI want to see cinder logs08:10
*** ajg20 has quit IRC08:12
noonedeadpunkhm, that's good question. I really thought they should be in /var/log/lxc, but don't see them there... So checking it as well08:12
noonedeadpunkI guess they're in journald as well...08:12
gokhaninoonedeadpunk, I find the error. It configure ceph.conf wrong on rgw containers. it creates 2 section on cepph.conf like [client.rgw.test-infra1-ceph-rgw-container-26ac4587.rgw0] and [client.radosgw.test-infra1-ceph-rgw-container-26ac4587] and it adds keystone variable under the [client.radosgw.test-infra1-ceph-rgw-container-26ac4587]. But it need08:13
gokhanito add them under the [client.radosgw.test-infra1-ceph-rgw-container-26ac4587]08:13
gokhani*But it need to add them under the [client.radosgw.test-infra1-ceph-rgw-container-26ac4587.rgw0]08:14
noonedeadpunkIt actually depends on the systemd name of the service08:14
gokhaniI don't know this is ceph-ansible bug08:14
noonedeadpunkin OSA we have exactly that override (with rgw0) as default one08:14
noonedeadpunkso that's kind of why I asked you regarding need of the override08:15
recycleheronoonedeadpunk: thanks atleast there are some logs in the lxc container itself with journalctl08:16
noonedeadpunkso, um, you needed service logs inside container or about lxc service itself?08:17
noonedeadpunkie your container does not start and you whant to know why?08:18
noonedeadpunkas all service logs inside container are in journalctl08:18
noonedeadpunkgokhani: https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/ceph-rgw.yml#L308:18
noonedeadpunkso I think it's expected thing from ceph-ansible side08:19
recycleheronoonedeadpunk: I was expecting to see cinder_volume service logs on the lxc host located at /var/log/lxc/cinder_colume..... but maybe I was wrong and maybe those logs are something else and are related to the container creation. cuz they are dated back to nov21. I think thats when I launched.08:21
noonedeadpunkrecyclehero: yeah, all service logs are in journald nowadays08:23
gokhaninoonedeadpunk, gotcha:(   reason of overriding ceph.conf is following this guide > https://docs.openstack.org/openstack-ansible/latest/user/ceph/swift.html08:23
noonedeadpunkWould be great if you could suggest patch to clear the things once you get things working08:24
recycleheroI have this regarding nfs permission on nfs logs08:26
recycleherohttps://dpaste.com/6HJN4JUNS08:26
recycleheroI dont know what was my previous config08:26
recycleheronow its root:root 64408:26
gokhaninoonedeadpunk, ok firstly ı will test up to date variables08:28
*** gokhani has quit IRC08:31
*** gokhani has joined #openstack-ansible08:37
*** fridtjof[m] has quit IRC08:41
*** manti has quit IRC08:41
*** shyamb has quit IRC08:48
*** fridtjof[m] has joined #openstack-ansible08:52
ptoI know its a little OT, but have anyone here used rbd export or rbd export-diff?08:57
noonedeadpunkI was only thinkinking about it's usage for some scenarios, but never really did09:07
*** priteau has joined #openstack-ansible09:22
*** manti has joined #openstack-ansible09:22
noonedeadpunkhm, what's wrong with journald log collection https://zuul.opendev.org/t/openstack/build/cd03de10a58848dfad282a8a1843c388/log/job-output.txt#2671009:42
noonedeadpunkum, `capture_output` is py38 only?09:43
*** tosky has joined #openstack-ansible09:45
noonedeadpunk3.7+09:46
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Make journal_dump py3.6 compatable  https://review.opendev.org/c/openstack/openstack-ansible/+/78846509:55
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: [goal] Deprecate the JSON formatted policy file  https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/78224409:56
ptonoonedeadpunk: I need to migrate an old ceph cluster to a new one, and i have been playing with rbd export and rbd export-diff - the performance is horrible slow10:10
openstackgerritGökhan proposed openstack/openstack-ansible master: Update ceph.conf for OpenStack-RadosGW integration  https://review.opendev.org/c/openstack/openstack-ansible/+/78847010:44
*** pcaruana has quit IRC10:52
*** pcaruana has joined #openstack-ansible11:02
admin0i could not find the rocky (16) -> stein (18) upgrade nodes .. if anyone has it , can you please pass it.. so that i can save those11:23
admin0gokhani, when you have it working, please pass me the notes as well .. i have ceph with rgw and osa .. still unclear on how exactly to add endpoint in keystone for it to work like swift11:24
noonedeadpunkyeah, in docs they're broken for some reason :/11:26
admin0there was also a etherpad link i think11:27
gokhaniadmin0, yes it is working now. OSA itself adds endpoints in https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/ceph-rgw-install.yml#L1611:35
admin0gokhani, you used osa+ceph playbook ( single playbook ) or  osa separately and ceph-ansible separately11:36
gokhaniadmin0, I used osa+ceph not seperately11:36
gokhaniadmin0 there is a bug in https://docs.openstack.org/openstack-ansible/latest/user/ceph/swift.html. If you want to use S3 api also change "client.rgw.{{ hostvars[inventory_hostname]['ansible_hostname'] }}" to "client.rgw.{{ hostvars[inventory_hostname]['ansible_hostname'] }}.rgw0"11:42
admin0gokhani, i have a few clusters with osa separate and ceph separate ..  if you can share me your openstack service list and openstack endpoint list ( only relevant to s3/swift/ceph) .. that would help11:43
admin0i would know how to add it to keystone11:43
gokhaniadmin0 , http://paste.openstack.org/show/804824/11:46
admin0gokhani, thank you .. with this, i can try to add it and see how it goes11:47
admin0noonedeadpunk, if you have the xenial -> bionic nodes url at hand, can you pass it to me once11:48
*** rh-jelabarre has joined #openstack-ansible11:49
gokhaniadmin0, ok good luck:)  If you need I can also share my ceph.conf.11:50
noonedeadpunkah, you meant X->B upgrade11:56
noonedeadpunkhttps://docs.openstack.org/openstack-ansible/rocky/admin/upgrades/distribution-upgrades.html11:57
noonedeadpunkBut I think I dropped etherpad from favorites once this got merged11:58
noonedeadpunkhttps://etherpad.openstack.org/p/osa-rocky-bionic-upgrade11:59
noonedeadpunkfound it11:59
openstackgerritGökhan proposed openstack/openstack-ansible master: Update ceph.conf for OpenStack-RadosGW integration  https://review.opendev.org/c/openstack/openstack-ansible/+/78847012:18
gokhaninoonedeadpunk ^^ these are for OpenStack - RadosGW integration. these confs are working.12:21
noonedeadpunkgokhani: can we leave `_`? I'm 99,9% believe that it's super fine having `rgw_keystone_url`12:22
noonedeadpunk(and etc)12:23
noonedeadpunkor can you check this out?12:24
gokhaninoonedeadpunk, yes this definition (with '_') is also working. I only updated them according to ceph docs https://docs.ceph.com/en/latest/radosgw/keystone/12:25
admin0c3 = primary host ( vip is there)  ; c3 repo = /etc/lsyncd/lsyncd.conf.lua .. the step says "disable non primary repo all-back" -- means I can reinstall c1 and c2 ( as its not primary) and then because repo is on c3, i need to disable c1 and c2 repo conainers ?12:25
*** gokhani has quit IRC12:28
*** gokhani has joined #openstack-ansible12:28
noonedeadpunkgouthamr: would you mind a bit editing your patch?12:29
gouthamrnoonedeadpunk: hey! Which one?12:30
noonedeadpunkgouthamr: sorry, meant gokhani, but he just left :(12:31
noonedeadpunkfalse ping12:31
gouthamrah, np12:31
*** gokhani has quit IRC12:31
*** gokhani has joined #openstack-ansible12:32
gokhaninoonedeadpunk , yes I can edit12:32
noonedeadpunkjust posted comment12:36
openstackgerritGökhan proposed openstack/openstack-ansible master: Update rgw client definition for OpenStack-RadosGW integration  https://review.opendev.org/c/openstack/openstack-ansible/+/78847012:38
noonedeadpunkgokhani: can you check comments in the gerrit?:)12:39
gokhaninoonedeadpunk , I changed it with previous _:(  I now see your comments12:39
noonedeadpunkyou can leave _ - whatever12:39
mgariepyanyone knows if the ephemeral disk filename is supposed to be stored in the DB?12:40
noonedeadpunkI just realized it might be good idea to just inlcude these group_vars12:40
noonedeadpunkI think it's not?12:41
mgariepyi have an old cloud and for some reason out of 120 vms 4 have the ephemeral disk names `disk.local` instead of `disk.eph0`12:41
noonedeadpunkBut it's supposed to be stored in file...12:42
mgariepyyes12:42
mgariepyi know but the filename is kinda just weird.12:42
mgariepyi don't get why.12:42
mgariepysame filetype and all.12:42
noonedeadpunkah, it's not really ephemeral I guess12:44
mgariepyyes it's a qcow file on the compute.12:44
noonedeadpunkI think it's just not volume based instances?12:44
mgariepyyes12:45
mgariepyusing local raid0 drives..12:45
noonedeadpunkI mean it's kind of expected naming by nova12:46
noonedeadpunkhttps://opendev.org/openstack/nova/src/branch/master/doc/source/reference/block-device-structs.rst#libvirt-driver-specific-bdm-data-structures12:46
noonedeadpunk`The flavor-defined ephemeral disk` will be `disk.local`12:46
openstackgerritGökhan proposed openstack/openstack-ansible master: Update rgw client definition for OpenStack-RadosGW integration  https://review.opendev.org/c/openstack/openstack-ansible/+/78847012:47
noonedeadpunkbtw `disk_info` is not really reliable and I don't think it has real effect onsmth12:47
mgariepywell i am wondering when i migrate the vms with the block devices if it will break :D12:48
mgariepyall other vms with a similar setup do have a disk.eph0 not disk.local :/12:48
noonedeadpunkI think we have most instances with jsut `disk` and super rarely with disk.eph012:49
mgariepyyeah the vda/root disk is disk.12:49
mgariepybut i do have falvors with epehemeral disk to be used as local scratch space (or more like persistent storage until one of the multiple drive fail)12:50
*** johanssone has quit IRC12:51
mgariepywell. it's supposed to be ephemeral anyway. so. let's see how it goes ..12:52
noonedeadpunkyeah sorry, dunno....12:53
mgariepywell it's only 4 vms.12:53
mgariepyand most likely only one.. if it fail badly i'll dig a bit more ..12:53
*** johanssone has joined #openstack-ansible12:54
jrosseradmin0: the OSA playbooks should add the radosgw stuff to keystone even for external ceph integration12:55
openstackgerritGökhan proposed openstack/openstack-ansible master: Update rgw client definition for OpenStack-RadosGW integration  https://review.opendev.org/c/openstack/openstack-ansible/+/78847012:56
*** spatel_ has joined #openstack-ansible12:56
*** spatel_ is now known as spatel12:56
*** pto has quit IRC13:05
*** macz_ has joined #openstack-ansible13:05
openstackgerritGökhan proposed openstack/openstack-ansible master: Update rgw client definition for OpenStack-RadosGW integration  https://review.opendev.org/c/openstack/openstack-ansible/+/78847013:11
openstackgerritGökhan proposed openstack/openstack-ansible master: Update rgw client definition for OpenStack-RadosGW integration  https://review.opendev.org/c/openstack/openstack-ansible/+/78847013:14
*** johanssone has quit IRC13:21
*** pto has joined #openstack-ansible13:26
*** johanssone has joined #openstack-ansible13:28
*** pto has quit IRC13:30
*** LowKey has quit IRC13:44
*** LowKey[A] has joined #openstack-ansible13:45
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Use neutron_conf_dir for absent policy removal  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/78849914:17
*** dave-mccowan has joined #openstack-ansible14:36
dmsimardnoonedeadpunk, sshnaidm: you both mentioned roles in the openstack collection from PTG discussions, out of curiosity I was wondering what kind of roles you had in mind ?14:43
dmsimardit's worth considering that since roles will be in the collection, they would also ship out of the ansible community package (ansible on pypi) so it could open opportunities14:44
sshnaidmdmsimard, some useful roles that use modules and will be easier to consume them, also will server as module usage example14:44
sshnaidms/server/serve/14:44
sshnaidmdmsimard, if you know "terraform modules", so something like that14:44
dmsimardso say, a role to create a server for example ? taking care of creating a flavor, a ssh key, the instance, security groups and stuff ?14:44
sshnaidmdmsimard, yep, most common scenarios, easy to configure14:45
dmsimardmakes sense, thanks14:45
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: DNM Change task ordering to perform smooth upgrades  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/78850114:49
noonedeadpunkjrosser: I feel like we need smth like that for all roles ^14:54
noonedeadpunkbecause right now we symlink empty directory at the beginning of the play in pre_tasks14:56
noonedeadpunkso all policy overrides and rootwrap is kind of broken till service restart14:56
noonedeadpunknot 100% sure in solution yet...14:58
* recyclehero back from work15:09
recycleherothats what happend from my instances working to some of them now booting in grub rescue:15:10
recycleherodamaged raid array of the nfs share which holds instances volumes - copied volumes to external hdd- created new array- did some new filesystem tuning to the raid array(stripe-width)- copied back from the external hdd15:12
*** gokhani has quit IRC15:29
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: DNM Change task ordering to perform smooth upgrades  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/78850115:30
*** d34dh0r53 has quit IRC15:48
*** d34dh0r53 has joined #openstack-ansible15:59
recycleheroI am seeing this repeat in cinder-volume container logs16:08
recycleheroApr 28 20:37:52 infra1-cinder-volumes-container-abe2b792 sudo[5307]:   cinder : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/openstack/venvs/cinder-21.1.0/bin/cinder-rootwrap /etc/cinder/rootwrap.conf du --bytes -s /var/lib/cinder/mnt/39ae8d6d2110507a51bef6f9f4c6f5ab16:08
jrosserrecyclehero: that doesnt necessarily look bad - it's checking the free space?16:16
recycleherojrosser: so 39a...5ab is name openstack gave to my nfs share16:17
recyclehero?16:17
jrosseri expect so16:17
jrosseras a first guess thats the ID of the volume16:18
recycleherojrosser:  its a dir and contains all the volumes16:19
jrosserok, cool :)16:19
jrosseri feel i'm sightly missing the point here somewhere - the log line you pasted was checking the free space peridocally, i'm thinking that this is normal16:20
recycleherothe big point is some of my valuable instances wont boot :((16:22
recycleheroI was able to create a volume in the nfs share16:22
jrosseroh sorry, you asked about an cinder log line and i answered :/16:22
*** d34dh0r53 has quit IRC16:22
recycleherobut when I wanted to launch an instance with that volume I got this16:23
recycleheroApr 28 20:50:44 infra1-cinder-volumes-container-abe2b792 cinder-volume[390]: 2021-04-28 20:50:44.486 390 WARNING py.warnings [req-63ace45e-e16f-4fb9-a11b-d999f2b0886e cc7d1a7f201f4b01b6a06e4923bd0805 ebe4a735e8d847f9ba37518db854fd1f - default default] /openstack/venvs/cinder-21.1.0/lib/python3.7/site-packages/sqlalchemy/orm/evaluator.py:99: SAWarning: Evaluating non-mapped column expression16:23
recyclehero'updated_at' onto ORM instances; this is a deprecated use case.  Please make use of the actual mapped columns in ORM-evaluated UPDATE / DELETE expressions.16:23
recyclehero "UPDATE / DELETE expressions." % clause16:23
recycleherois it okay to paste one log lines here?16:23
jrosserpaste.openstack.org really16:23
recycleherosure16:24
recycleherohttp://paste.openstack.org/show/804838/16:27
recycleheroi bet this is fimiliar to some of you. whats wrong with my volumes?16:27
*** d34dh0r53 has joined #openstack-ansible16:30
jrossernot sure really, but you might see if things look consistent between openstack server show <server_id> and openstack volume show <volume_id>16:35
jrosserlike if there are volumes aparrently attched to your server but those don't show up on the cinder side, or vice versa16:35
*** rpittau is now known as rpittau|afk16:39
admin0so while redoing a controller from xenial -> bionic, setup hosts and infra went OK .. except ceph-mon is not able to join the cluster .. this is the error: https://gist.githubusercontent.com/a1git/1a3c883ec23bd9c2904333843ca7b80c/raw/2ef87a183bdcb0fe7639d7ccd2a063c54b0f54b2/gistfile1.txt  .. anyone seen this before and knows of it ?16:49
admin0https://etherpad.opendev.org/p/osa-rocky-bionic-upgrade mentions of nova failing due to newer packages16:50
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Change task ordering to perform smooth upgrades  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/78850117:00
*** gyee has joined #openstack-ansible17:39
admin02021-04-28 17:23:21.687538 7f68c16bb700 -1 mon.c2-ceph-mon-container-79f39d53@0(probing) e0 handle_probe missing features, have 4611087853746454523, required 0, missing 0 is the error in the new container .. the only reference i could find was in a redhat  and the solution said to use the same package for ceph17:40
admin0so right now, c1 and c3 are in xenial .. c2 in bionic17:40
*** andrewbonney has quit IRC18:00
jrosseradmin0: have you done basic connectivity checks from the new to old mon?18:06
jrosser"required 0" is very suspicious18:06
openstackgerritMerged openstack/openstack-ansible-os_neutron master: Updated from OpenStack Ansible Tests  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/78685518:07
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Change task ordering to perform smooth upgrades  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/78850119:16
*** dave-mccowan has quit IRC20:22
*** dave-mccowan has joined #openstack-ansible20:26
*** spatel has quit IRC21:43
*** rh-jelabarre has quit IRC21:45
*** macz_ has quit IRC22:42
*** tosky has quit IRC22:48
*** luksky has quit IRC22:56
*** kleini has quit IRC23:27

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!