Thursday, 2020-04-02

*** gyee has quit IRC00:08
*** DanyC has joined #openstack-ansible00:20
*** DanyC has quit IRC00:24
*** DanyC has joined #openstack-ansible01:17
*** DanyC has quit IRC01:23
*** weshay|ruck is now known as weshay01:31
*** jamesden_ has joined #openstack-ansible01:34
*** jamesden_ has quit IRC01:42
*** evrardjp has quit IRC04:36
*** evrardjp has joined #openstack-ansible04:37
*** udesale has joined #openstack-ansible04:56
*** threestrands has joined #openstack-ansible05:06
*** thuydang has quit IRC05:14
*** kleini has joined #openstack-ansible05:24
*** thuydang has joined #openstack-ansible05:30
snadgei still cant figure out why neutron_server-back keeps failing.. this is saddening, it wasn't doing this before06:14
*** miloa has joined #openstack-ansible06:17
*** spatel has joined #openstack-ansible06:20
*** spatel has quit IRC06:25
snadgei wish i knew more about this stuff, ive poked around in rabbitmq .. can see "AMQP server on 172.29.237.199:5672 is unreachable: timed out." in the neutron logs06:49
snadgein the rabbit logs, i can see "client unexpectedly closed TCP connection" for the neutron user.. i've tried disabling SSL and connecting to port 5672 instead of 5671.. just to see if that might make a difference, apparently not06:52
*** gary_perkins has quit IRC06:55
*** gary_perkins has joined #openstack-ansible06:58
*** jbadiapa has joined #openstack-ansible07:01
*** DanyC has joined #openstack-ansible07:15
*** rpittau|afk is now known as rpittau07:17
*** fridtjof[m] has quit IRC07:29
*** masterpe has quit IRC07:29
*** gokhani has joined #openstack-ansible07:34
noonedeadpunkmnaser: I see you don't like suse, but as openstack governance claims that suse is supported, and we already have it, not sure we should drop it. Also, it do not explain why it passes tests and fails _only_ in gates. Or we're going to drop all jobs that fails in gates?07:42
noonedeadpunkI think that here's smth weird stuff07:42
noonedeadpunkalso currently we have hardening kinda broken, so maybe it's not only suse fault....07:44
jrossersnadge: have you enabled the rabbitmq monitoring dashboard?07:46
noonedeadpunkmnaser: if suse would be dropped out of governance ofc I see no reason in it's futher support07:47
jrosseralso check the cpu usage of your rabbit processes, if I had to guess I’d say you have either runaway generation of excess rabbit queues or in-acked messages sat in rabbit building up07:48
*** fridtjof[m] has joined #openstack-ansible07:59
*** threestrands has quit IRC08:01
*** DanyC has quit IRC08:01
*** tosky has joined #openstack-ansible08:02
*** chigang_ has joined #openstack-ansible08:03
jrosserchandankumar: https://review.opendev.org/#/c/710251/ wont pass by rechecking08:07
jrossertheres a bug we need to fix08:07
chandankumarjrosser, ok08:07
chandankumarthanks for pointing out08:07
*** fghaas has joined #openstack-ansible08:14
*** cp- has quit IRC08:21
*** cp- has joined #openstack-ansible08:23
*** cp- has quit IRC08:23
*** cp- has joined #openstack-ansible08:24
*** masterpe has joined #openstack-ansible08:42
snadgei managed to create an admin user and have logged into the web interface via a port forward..and it all seems pretty normal08:50
jrosseryou should be able to see if there are messages lying aroung08:50
snadgeunacked builds up to about 30.. over the space of a few minutes, then drops to zero08:52
snadgethey're all in q-reports-plugin08:53
*** fghaas has quit IRC08:56
snadgetrying to figure out what that does and whether i can disable it09:03
*** DanyC has joined #openstack-ansible09:03
*** fghaas has joined #openstack-ansible09:11
*** andrewbonney has joined #openstack-ansible09:25
*** sshnaidm|afk is now known as sshnaidm09:25
snadgeat least im a step closer.. i just need to figure out what actually consumes those messages, and why its taking so long09:29
*** ygk_12345 has joined #openstack-ansible09:52
gokhaniHi team ı am trying to add ironic my OpenStack train (20.0.0 tag) Environment , but it gives error when install pip packages in venv. it seems it is parsing error. openstack-ansible os-ironic_install.yml output is http://paste.openstack.org/show/791512/ and python venv build log is http://paste.openstack.org/show/791511/09:57
gokhaniı didn't get which requirement gives error09:58
ygk_12345I am facing this error while upgrading from rocky to stein manually after this command "${SCRIPTS_PATH}/bootstrap-ansible.sh"  . check this error  http://paste.openstack.org/show/791514/10:00
ygk_12345can someone help me with that error p[lease10:00
*** spatel has joined #openstack-ansible10:09
noonedeadpunkygk_12345: hm, I'm pretty sure that the mentioned commit exist on github... https://github.com/logan2211/ansible-resolvconf/commit/a2ff5ba59b47f96ddddcb7a3a67de93687c317a610:11
ygk_12345noonedeadpunk so whats the workaround now ?10:12
*** spatel has quit IRC10:14
ygk_12345noonedeadpunk can you please hep me  how to proceed now10:14
*** rpittau is now known as rpittau|bbl10:14
noonedeadpunkygk_12345: can you try going to /etc/ansible/roles/ansible-resolvconf, doing git pull; git checkout a2ff5ba59b47f96ddddcb7a3a67de93687c317a6 ?10:14
noonedeadpunkgokhani: I can recall that we had some bug with ironic on 20.0.010:15
noonedeadpunkiirc it has been fixed on 20.0.110:15
noonedeadpunkbut not sure if it was this exact issue....10:16
ygk_12345noonedeadpunk I have deleted the resolvconf folder in /etc/ansible/roles  and rerunning the script10:17
noonedeadpunkor that10:17
gokhaninoonedeadpunk, ok ı am trying to find this commit10:19
noonedeadpunkI guess it's https://review.opendev.org/#/c/709747/10:20
gokhaninoonedeadpunk, ok ı am tryin now this commit10:22
*** gshippey has joined #openstack-ansible11:08
ebbexI'm having some issues with live-migration. I've found several inconsistencies in the database tables, and wondering which connection could be problematic when I'm seeing these in the logs;11:18
ebbex[instance: 0a4a2a9b-a6d1-491f-b59a-d9eca7ed6d91] Detaching volume from unknown instance11:18
ebbex[instance: 0a4a2a9b-a6d1-491f-b59a-d9eca7ed6d91] During detach_volume, instance disappeared.: InstanceNotFound: Instance 0a4a2a9b-a6d1-491f-b59a-d9eca7ed6d91 could not be found.11:18
ebbexThis is happening on the target hypervisor. Where the instance is moved with "openstack server migrate --live target-hypervisor 0a4a2a9b-a6d1-491f-b59a-d9eca7ed6d91"11:19
ebbex[instance: 0a4a2a9b-a6d1-491f-b59a-d9eca7ed6d91] Attempting to driver detach volume b134615d-70ae-41b3-94f7-a44dfc9fd7f6 from mountpoint /dev/vda11:20
ebbexI suspected some values were wrong in nova_cell1.block_device_mapping, or cinder.volumes. But don't see any wrong connections there.11:21
ebbexOnly under cinder.volumes the 'host' column containing something like "compute01@RBD#rbddriver", but those values don't add up for any of the vms and the compute-hosts they're running on.11:25
*** DanyC has quit IRC11:26
*** ygk_12345 has quit IRC11:40
*** udesale_ has joined #openstack-ansible11:40
*** udesale has quit IRC11:43
*** DanyC has joined #openstack-ansible11:53
noonedeadpunkmnaser: jrosser: seems it was broken suse gate image that got upadet as suse passed hardening11:56
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-hardening master: [ussuri][goal] Updates for python 2.7 drop  https://review.opendev.org/71059812:10
*** rpittau|bbl is now known as rpittau12:12
*** ygk_12345 has joined #openstack-ansible12:14
ygk_12345hi all I am following this rocky-stein upgrade https://docs.openstack.org/openstack-ansible/stein/admin/upgrades/major-upgrades.html  process. after running the setup-hosts command I see another set of containers being created on the hosts. is this normal and accepted behaviour ?12:15
ygk_12345I see two containers of each type now on the hosts ?12:16
ygk_12345is this the accpeted behaviour ?12:16
ebbexygk_12345: from rocky to stein, i don't think so. didn't happen on my upgrades.12:21
ygk_12345ebbex i see that the rocky version containers are not deleted or removed and the the new stein containers are created12:23
jrosserthe upgrade should not create new containers12:26
ygk_12345jrosser but it did in mycase12:27
ebbexhow does your /etc/openstack_deploy/openstack_inventory.json look, any "duplicate" containers there? e.g. two infra01_cinder_api containers?12:28
ebbexygk_12345: I'm gonna ask a dumb question, but you had the old inventory in place right?12:29
ygk_12345ebbex ahhh yes12:30
ygk_12345i see all duplicates in the inventroy now12:31
ygk_12345ebbex sorry no duplicates in the inventroy. my mistake12:31
ygk_12345the inventory has old continaers12:31
ygk_12345should I delete them first prioir to upgrade12:32
ygk_12345?12:32
gokhaniygk_12345, ı think this link help you for removing duplicate containers from inventory: https://docs.openstack.org/openstack-ansible/latest/reference/inventory/manage-inventory.html12:32
ygk_12345ebbex confused now a bit12:33
ebbexNope, if they are there it the upgrade should reuse them, if they were missing, it'd just create random new ones as you're experiencing.12:33
ygk_12345ebbex oh no i think I have to do it from scratch12:33
ygk_12345rocky---stein again12:33
ebbexYeah, the big fat red warning on that page summarises what I was asking/checking.12:34
ebbexAnd from the sound of your situation seemed to be the likely cause of what you were experiencing.12:35
*** fghaas has left #openstack-ansible12:38
ygk_12345ebbex i have deleted the duplicated contianers. how do I repopulate the inventry.json with the old list ?12:41
ygk_12345with the existing continaers ?12:42
mnasernoonedeadpunk: it's a chicken and egg wrt suse, we don't "have" to support them because they're in the tc list12:42
mnaserfor example, i dont thin kolla has support, nor tripleo12:43
openstackgerritMarcus Klein proposed openstack/openstack-ansible-os_keystone master: Use the service setup host python interpreter for LDAP domain creation  https://review.opendev.org/71697112:43
openstackgerritMarcus Klein proposed openstack/openstack-ansible-os_keystone master: Same service setup host python interpreter for LDAP domains  https://review.opendev.org/71697112:46
openstackgerritMarcus Klein proposed openstack/openstack-ansible-os_keystone stable/stein: Same service setup host python interpreter for LDAP domains  https://review.opendev.org/71697312:49
openstackgerritMarcus Klein proposed openstack/openstack-ansible-os_keystone stable/train: Same service setup host python interpreter for LDAP domains  https://review.opendev.org/71697512:50
snadgei think two of my compute nodes might be flakey, which causes enough lag in the neutron-server process for it to drop out, it only has a couple of threads allocated to it12:53
snadgeso im just going to turn them all on except the ones im suspicious of and see if it copes with that.. then turn them on one at a time.. then both.. and possibly tweak the number of processes etc so its at least equal to the number of compute nodes, so they can be done in parallel without bunching up and causing timeouts12:56
*** ygk_12345 has quit IRC12:57
kleininoonedeadpunk: https://review.opendev.org/#/c/716971/2 <- please see my submit of the necessary fix for me.12:57
noonedeadpunkkleini: we ususally cherry-pick to stable branches once master one merges13:08
noonedeadpunkbut leave cherry-pick as is as they'r created - it's just for the future:)13:08
jrossersnadge: i think there may be issues with threads vs processes too, though i don't remember which component specifically13:15
snadgeit looks like i might have to experiment with it13:15
jrosserafaik some bits of openstack when used with wsgi shouldn't also use threads13:15
*** ianychoi_ has joined #openstack-ansible13:17
*** ianychoi has quit IRC13:20
kleinihups, sorry. should have asked for the process.13:21
*** ianychoi_ is now known as ianychoi13:23
*** jamesden_ has joined #openstack-ansible13:25
snadgeshould shutting a compute node down also remove its hypervisor entry?13:25
snadgeor do they get stuck sometimes13:26
snadgeive seen some stuff online which suggests going into the database and manually removing the entry13:26
gokhanihi folks, ı am trying to add ironic to my OpenStack Environment, but when ı try to install with openstack-ansible, I see that ı also need swift. ı am sorry may be this is a silly question I am new to ironic , is swift is a requirement for ironic ? ıs there any alternative to swift in ironic installation ?13:33
*** xakaitetoia has joined #openstack-ansible13:34
openstackgerritMerged openstack/openstack-ansible stable/train: Fix python2 support  https://review.opendev.org/71521513:51
*** weshay is now known as weshay|ruck13:55
*** thuydang has quit IRC14:02
jrossergokhani: i've never used ironic but just looking at the ansible role it would seem that you can run it without an object store https://github.com/openstack/openstack-ansible-os_ironic/blob/master/templates/ironic.conf.j2#L5914:07
gokhanijrosser, thanks ı will try ironic standalone14:17
*** sshnaidm is now known as sshnaidm|afk14:25
snadgei wonder why this one particular blade causes neutron-server to fall over, for now i've disabled it.. i can use the playbook to remove it if i need to14:26
snadgeits working fine with 7 of 8 enabled though, so i'll just leave it at that, maybe swap it out for another one or run some tests on it14:26
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-hardening master: [ussuri][goal] Updates for python 2.7 drop  https://review.opendev.org/71059814:31
jrosserhrrm another keystone blow up https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_95a/716971/2/check/openstack-ansible-deploy-aio_metal-debian-buster/95a87c1/logs/host/keystone-wsgi-public.service.journal-14-14-03.log.txt14:44
*** macz_ has joined #openstack-ansible14:53
*** macz_ has quit IRC14:55
*** macz_ has joined #openstack-ansible14:55
noonedeadpunkand for buster only?14:59
noonedeadpunkpy3.7 thing14:59
noonedeadpunk(probably)15:00
noonedeadpunkjrosser: do you see why this got skipped? https://zuul.opendev.org/t/openstack/build/a6af7880a78e42e0bff352f34275d8fe/log/job-output.txt#287015:00
noonedeadpunkoh, it's run with --check15:02
jrosserthe one from yesterday was 3.6 https://zuul.opendev.org/t/openstack/build/dc90c65b0e194aaaba56cfc8c2ed5ac5/log/logs/host/keystone-wsgi-public.service.journal-19-03-39.log.txt#460815:03
noonedeadpunkbut reasons seem different. this one because of failed memcached daemon15:04
*** DanyC has quit IRC15:07
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/train: Bump SHAs for stable/train  https://review.opendev.org/71431015:07
*** DanyC has joined #openstack-ansible15:07
noonedeadpunkcores, can we vote on this? ^15:07
*** DanyC has joined #openstack-ansible15:08
*** DanyC has joined #openstack-ansible15:09
*** DanyC has joined #openstack-ansible15:09
*** DanyC has joined #openstack-ansible15:10
*** DanyC has joined #openstack-ansible15:10
*** gyee has joined #openstack-ansible15:11
*** DanyC has joined #openstack-ansible15:13
*** sshnaidm|afk is now known as sshnaidm15:15
*** DanyC has joined #openstack-ansible15:18
*** ondrejburian has quit IRC15:19
mgariepynoonedeadpunk, done,15:22
noonedeadpunkthanks:)15:22
mgariepynoonedeadpunk, any other that needs my attention ?15:23
noonedeadpunkmaybe https://review.opendev.org/#/q/topic:use-integrated-repo-for-all-tests+status:open ? :)15:24
*** gokhani has quit IRC15:26
noonedeadpunk🖒15:29
mgariepylol you are now doing in emoji ? haha15:29
noonedeadpunkLearning ascii codes lol15:29
mgariepyhaha :D15:30
mgariepyalways useful.15:30
noonedeadpunkyeah :)15:31
*** jamesden_ has quit IRC15:31
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-hardening master: [ussuri][goal] Updates for python 2.7 drop  https://review.opendev.org/71059815:40
*** ondrejburian has joined #openstack-ansible15:42
*** rpittau is now known as rpittau|afk15:56
jrossermnaser: seems to be a huge pile of new work in the mitogen repo16:13
jrosseri might have another spin with that as i think some of our issues may be addressed16:13
noonedeadpunkI guess we're talking about it's usage for years...16:14
jrosseryes16:15
jrossersomeone is doing serious work on the code now16:15
noonedeadpunkoh, yeah, I see16:20
noonedeadpunkrealy lots of activity16:20
mnaseroh nice16:23
*** udesale_ has quit IRC16:26
*** evrardjp has quit IRC16:36
*** evrardjp has joined #openstack-ansible16:36
*** DanyC has quit IRC16:51
*** DanyC has joined #openstack-ansible16:58
*** KeithMnemonic has joined #openstack-ansible17:28
*** cosmicsound has joined #openstack-ansible17:38
*** xakaitetoia has quit IRC17:38
*** xakaitetoia has joined #openstack-ansible17:48
*** xakaitetoia has quit IRC17:56
*** DanyC has quit IRC18:10
*** DanyC has joined #openstack-ansible18:13
*** DanyC has joined #openstack-ansible18:17
*** DanyC has joined #openstack-ansible18:17
openstackgerritMerged openstack/openstack-ansible-os_zun master: Drop commented out values from configs  https://review.opendev.org/69354318:23
*** xakaitetoia has joined #openstack-ansible18:24
*** sshnaidm is now known as sshnaidm|afk18:24
*** xakaitetoia has quit IRC18:29
jrosseron this mitogen AIO its very obvious the differentce between ansible forks 5 and 1018:33
jrosserit's clearly doing two batches of 5 with a perceptible delay between for 5, and with 10 all the containers get done in parallel18:34
noonedeadpunkoh, rly? sounds really good18:35
jrosseri wonder if there is good reason to just up forks = n_containers for AIO anyway18:35
*** miloa has quit IRC18:37
noonedeadpunkI guess number of cores in CI flavor allows us to do that18:43
noonedeadpunkActually I think the most heavy one will be telemetry stack18:43
noonedeadpunk(in terms of containers #)18:43
jrosserthe container create is cpu bound on one core unzipping the base image too18:44
jrossertar --anchored --numeric-owner -xpJf /var/cache/lxc//download/ubuntu/bionic/amd64//default/rootfs.tar.xz -C /var/lib/lxc/aio1_placement_container-130e2a4a/rootfs18:45
noonedeadpunkI thought you was talking about roles runtime?18:45
jrosseryes sorry i am for ansible forks18:46
noonedeadpunkyeah, for containers creation I think there's liitle we can do...18:47
noonedeadpunkbut maybe we will fix this ansible slow downs with grown # of hosts...18:47
noonedeadpunkActually I didin't test moving a-r-r to group_vars yet:(18:47
jrosseri should spend some time just looking at what we have in hostvars18:50
jrossersee how much unnecessary stuff we are copying around18:50
*** DanyC has quit IRC18:54
openstackgerritMerged openstack/ansible-role-systemd_networkd stable/train: Updated from OpenStack Ansible Tests  https://review.opendev.org/69038418:58
*** xakaitetoia has joined #openstack-ansible19:01
openstackgerritMerged openstack/ansible-role-systemd_service stable/train: Updated from OpenStack Ansible Tests  https://review.opendev.org/69038519:02
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Add zun service to tempest  https://review.opendev.org/69240619:03
*** xakaitetoia has quit IRC19:07
openstackgerritMerged openstack/ansible-role-systemd_mount stable/train: Updated from OpenStack Ansible Tests  https://review.opendev.org/69038219:10
*** cosmicsound has left #openstack-ansible19:16
openstackgerritMerged openstack/openstack-ansible-os_trove stable/train: Updated from OpenStack Ansible Tests  https://review.opendev.org/69037819:22
*** xakaitetoia has joined #openstack-ansible19:34
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-hardening master: [ussuri][goal] Updates for python 2.7 drop  https://review.opendev.org/71059819:38
*** xakaitetoia has quit IRC19:39
*** Chaserjim has joined #openstack-ansible19:49
openstackgerritMerged openstack/openstack-ansible-os_murano stable/train: Updated from OpenStack Ansible Tests  https://review.opendev.org/71474519:51
noonedeadpunkmnaser: https://review.opendev.org/#/c/713511/ passing20:00
mnasernoonedeadpunk: nice, +2 .. is stable/train g2g so we can recheck stuff thats borked?20:06
noonedeadpunkYes20:06
*** DanyC has joined #openstack-ansible20:07
*** xakaitetoia has joined #openstack-ansible20:09
*** DanyC has quit IRC20:11
*** xakaitetoia has quit IRC20:17
mnasernoonedeadpunk: cool, in that case i'll do that soon20:18
openstackgerritMerged openstack/openstack-ansible stable/train: Set bionic upgrade job to voting  https://review.opendev.org/70823820:29
*** gshippey has quit IRC20:30
jrossertrain patches are rechecked20:35
*** jbadiapa has quit IRC20:42
*** weshay|ruck is now known as weshay20:57
*** kleini has quit IRC21:38
jrosser\o/ passed tempest with mitogen and containers22:03
jrossernever managed that before22:03
*** DanyC has joined #openstack-ansible22:31
*** DanyC has quit IRC22:36
openstackgerritMerged openstack/openstack-ansible-os_keystone master: Same service setup host python interpreter for LDAP domains  https://review.opendev.org/71697122:43
*** openstackstatus has quit IRC22:56
*** openstack has joined #openstack-ansible23:00
*** ChanServ sets mode: +o openstack23:00
*** tosky has quit IRC23:00
*** DanyC has joined #openstack-ansible23:07
*** DanyC has quit IRC23:13
*** mrda has quit IRC23:22
*** gyee has quit IRC23:38
openstackgerritMerged openstack/openstack-ansible master: Add support to deploy zun  https://review.opendev.org/69240723:42

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!