| *** dwilde has quit IRC | 00:04 | |
| *** tosky has quit IRC | 00:12 | |
| *** luksky has quit IRC | 00:15 | |
| *** spatel_ has joined #openstack-ansible | 00:36 | |
| *** spatel_ is now known as spatel | 00:36 | |
| *** rh-jelabarre has quit IRC | 00:44 | |
| *** spatel has quit IRC | 01:14 | |
| *** spatel_ has joined #openstack-ansible | 01:15 | |
| *** spatel_ is now known as spatel | 01:15 | |
| *** jamesdenton has quit IRC | 01:16 | |
| *** jamesden_ has joined #openstack-ansible | 01:17 | |
| *** spatel has quit IRC | 02:09 | |
| *** evrardjp has quit IRC | 03:33 | |
| *** evrardjp has joined #openstack-ansible | 03:33 | |
| *** jamesden_ has quit IRC | 04:54 | |
| *** jamesdenton has joined #openstack-ansible | 04:54 | |
| *** yasemind has joined #openstack-ansible | 05:22 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Fix venv cleanup for uwsgi venv https://review.opendev.org/c/openstack/openstack-ansible-ops/+/782907 | 07:17 |
|---|---|---|
| *** miloa has joined #openstack-ansible | 07:21 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Extend timeout for RPM verification https://review.opendev.org/c/openstack/ansible-hardening/+/782909 | 07:47 |
| *** jamesdenton has quit IRC | 07:56 | |
| *** jamesdenton has joined #openstack-ansible | 07:57 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Fix condition when to create backends https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/782911 | 07:59 |
| *** miloa has quit IRC | 08:01 | |
| *** luksky has joined #openstack-ansible | 08:04 | |
| *** andrewbonney has joined #openstack-ansible | 08:10 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts stable/rocky: Determine latest base image available https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/782734 | 08:11 |
| *** rpittau|afk is now known as rpittau | 08:13 | |
| jrosser | morning | 08:22 |
| jonher | morning | 08:22 |
| jrosser | 777384 really is somehow cursed, i hope it has better luck passing tests today | 08:23 |
| *** jbadiapa has joined #openstack-ansible | 08:31 | |
| noonedeadpunk | o/ | 08:36 |
| *** tosky has joined #openstack-ansible | 08:43 | |
| ebbex | noonedeadpunk, jrosser: any thoughts on why only xenial started working here? https://review.opendev.org/c/openstack/openstack-ansible-tests/+/782799 | 08:48 |
| ebbex | note i'm overriding the url to (seems to me) a valid image for all test-os'es. | 08:49 |
| jrosser | from the test results? | 08:49 |
| * jrosser seeing them all broken | 08:49 | |
| ebbex | Ah, yes they fail on other stuff later on. but in the beginning they failed on lxc-image. and only bionic still does. | 08:50 |
| jrosser | issues with tempest plugins could be that the versions are not pinned back to py2 supporting versions for those OS | 08:50 |
| jrosser | it's this https://zuul.opendev.org/t/openstack/build/fb5362e6304d4861aa0676e7eff2c68d/log/logs/host/lxc-cache-prep-commands.log.txt#296-300 | 08:52 |
| ebbex | aha, cause i found that strange that neutron/master passes on centos but fails for xenial. and pinned cinder fails on centos but passes xenial. really weird. | 08:52 |
| jrosser | the https repos thing is worked around on more recent branches like this https://github.com/openstack/openstack-ansible-tests/blob/master/zuul.d/playbooks/pre-gate-cleanup.yml#L21-L27 | 08:53 |
| jrosser | and similar in the openstack-ansible repo too, depending on which jobs it is | 08:53 |
| ebbex | jrosser: thanks, that's a great find! | 08:54 |
| ebbex | :) | 08:55 |
| jrosser | np :) | 08:55 |
| openstackgerrit | Ebbex proposed openstack/openstack-ansible-tests stable/rocky: Update get-pip url https://review.opendev.org/c/openstack/openstack-ansible-tests/+/782799 | 09:09 |
| *** sshnaidm|afk is now known as sshnaidm | 09:24 | |
| openstackgerrit | Adrien Cunin proposed openstack/openstack-ansible-ops master: Fixed venvs pattern to be more specific https://review.opendev.org/c/openstack/openstack-ansible-ops/+/782750 | 09:26 |
| noonedeadpunk | ebbex: btw, I've already cherry-picked https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/782734 :( | 09:27 |
| *** gokhani has joined #openstack-ansible | 09:34 | |
| noonedeadpunk | I want to get this landed and backported to V before creating new V tag as it's pretty major bug :( https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/782911 | 09:37 |
| *** amalrajgenocidex has joined #openstack-ansible | 09:40 | |
| *** shyamb has joined #openstack-ansible | 09:54 | |
| *** amalrajgenocidex has quit IRC | 09:59 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_swift master: Revert "split templates to work around configparser bug" https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/782956 | 10:09 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_swift master: Revert "split templates to work around configparser bug" https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/782956 | 10:11 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_swift master: Revert "split templates to work around configparser bug" https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/782956 | 10:16 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: Use ansible_facts[] instead of fact variables https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/780732 | 10:20 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/772052 | 10:21 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/780376 | 10:21 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/782314 | 10:21 |
| *** rodolof has joined #openstack-ansible | 10:27 | |
| openstackgerrit | Merged openstack/ansible-hardening master: Extend timeout for RPM verification https://review.opendev.org/c/openstack/ansible-hardening/+/782909 | 10:32 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Adding support of subnet_dns_publish_fixed_ip extension in ml2 plugin https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/772245 | 10:37 |
| jrosser | noonedeadpunk: i was just thinking again about the qemu processes overrides you mentioned yesterday, do we need a patch for that? | 10:38 |
| jrosser | there is this which we could extend with a default set? https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/defaults/main.yml#L494 | 10:40 |
| noonedeadpunk | yep, I used that to cover the issue... | 10:40 |
| noonedeadpunk | So I'm not sure about setting a default... | 10:41 |
| noonedeadpunk | it was actually mine question if we need a patch :) | 10:41 |
| noonedeadpunk | or maybe we should jsut have it documented... | 10:41 |
| jrosser | it seems like a really common case with a good sized ceph cluster | 10:42 |
| noonedeadpunk | yeah, agree. so we can set it in case driver is ceph? | 10:42 |
| noonedeadpunk | but that actually would be also the case if cinder volumes are ceph based (while nova ephemeral are not) | 10:43 |
| jrosser | it could be nova with rbd, or cinder with rbd | 10:43 |
| noonedeadpunk | oh, we have nova_cinder_rbd_inuse, right | 10:43 |
| noonedeadpunk | will make a patch then | 10:43 |
| jrosser | cool | 10:43 |
| jrosser | we were going to try to look again at the galera upgrade stuff | 10:44 |
| jrosser | seems really hard dealing with the old root user authentication setup + move to the new root+admin style | 10:44 |
| noonedeadpunk | we also super need to merge masters bump I think (which is also broken on upgrades in tempest) | 10:46 |
| noonedeadpunk | and I must spend time on playing with trove atm :( | 10:46 |
| noonedeadpunk | to get it finally deployed in prod | 10:46 |
| jrosser | oh really many failures to retrieve u-c on the master bump patch | 10:50 |
| noonedeadpunk | yeah, there was some issue yestarday with the mirrors I guess | 10:51 |
| noonedeadpunk | Btw I commented https://review.opendev.org/c/openstack/openstack-ansible/+/777990/15/playbooks/common-playbooks/cinder.yml (not sure if you saw) | 10:51 |
| jrosser | yeah reasonable | 10:52 |
| jrosser | -ENOTIME :( | 10:52 |
| noonedeadpunk | really same here :( | 10:53 |
| jrosser | oh no wait | 10:54 |
| jrosser | arent those tasks separated out to make sure fact gathering happens properly with --limit? | 10:54 |
| jrosser | or tags maybe, becasue of the tags: always | 10:54 |
| noonedeadpunk | yes, it was because of the tags. But we can put tags on pre tasks? | 10:55 |
| noonedeadpunk | I haven't really tested it, but I guess it should work | 10:55 |
| jrosser | also looking at how cinder_serial would affect things | 10:56 |
| noonedeadpunk | and that's fair comment... | 10:56 |
| noonedeadpunk | it actually might affect | 10:56 |
| jrosser | feels like right now we gather all facts for the whole group up front in a really simple way | 10:56 |
| noonedeadpunk | still don't need repo_all? | 10:57 |
| jrosser | no i can take that away | 10:57 |
| noonedeadpunk | but agree, didn't thought about serial | 10:57 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Gather minimal facts in common playbooks https://review.opendev.org/c/openstack/openstack-ansible/+/777990 | 10:58 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Fixed venvs pattern to be more specific https://review.opendev.org/c/openstack/openstack-ansible-ops/+/782750 | 11:00 |
| openstackgerrit | Ebbex proposed openstack/openstack-ansible-tests stable/rocky: Update get-pip url https://review.opendev.org/c/openstack/openstack-ansible-tests/+/782799 | 11:04 |
| ebbex | jrosser: https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/782707 ReadTimeoutError("HTTPSConnectionPool(host='opendev.org', port=443): Read timed out. ERROR: 502 Server Error: Proxy Error for url... | 11:09 |
| *** gokhani has quit IRC | 11:09 | |
| ebbex | any ideas why there's a proxy-error on xenial but not centos7? | 11:09 |
| noonedeadpunk | recheck? | 11:10 |
| noonedeadpunk | or its persistant? | 11:10 |
| noonedeadpunk | I just saw lots of infras issues yestarday | 11:10 |
| *** gokhani has joined #openstack-ansible | 11:11 | |
| ebbex | it was twice yesterday, but i'll give the recheck a go. strange it didn't have any issues with centos7. | 11:11 |
| noonedeadpunk | might be different regions | 11:12 |
| *** shyamb has quit IRC | 11:32 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_trove master: Retry on creating trove network https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/782814 | 11:35 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova master: Fix usage of tags https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/782796 | 11:36 |
| *** gokhani has quit IRC | 11:43 | |
| noonedeadpunk | doh 777384 failed again | 11:43 |
| *** rh-jelabarre has joined #openstack-ansible | 11:52 | |
| openstackgerrit | Ebbex proposed openstack/openstack-ansible-tests stable/rocky: Update get-pip url https://review.opendev.org/c/openstack/openstack-ansible-tests/+/782799 | 11:57 |
| ebbex | wow, this passed: https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/782707 | 11:58 |
| *** shyamb has joined #openstack-ansible | 11:59 | |
| jonher | "Failed to connect to opendev.org port 443" in various jobs, infra has issues with gitea/lb so that probably explains your issue before as well | 11:59 |
| openstackgerrit | Ebbex proposed openstack/openstack-ansible-tests stable/queens: Remove temporary override for rabbitmq https://review.opendev.org/c/openstack/openstack-ansible-tests/+/782987 | 12:03 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_gnocchi master: Fix gnocchi installation for new pip resolver https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/781513 | 12:05 |
| noonedeadpunk | btw I hope this will fix gnocchi instlllation without need to maintain local set of requirements | 12:05 |
| noonedeadpunk | at least worked locally | 12:05 |
| noonedeadpunk | ok, so. master bump fails because of the `pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')` for nova-scheduler o_O | 12:09 |
| noonedeadpunk | mariadb bug? | 12:09 |
| noonedeadpunk | disregard, probably it was during mariadb upgrade | 12:10 |
| noonedeadpunk | super weird. tempest fails with `Details: (TestServerBasicOps:test_server_basic_ops) Server 40ce5b33-d2d8-470e-b8c8-776c53efce8c failed to reach ACTIVE status and task state "None" within the required time (600 s). Current status: BUILD. Current task state: scheduling.` https://be7a0942ef49bf0b6abe-b6dab27f1b1aff436bdeb943180547ea.ssl.cf5.rackcdn.com/780434/8/check/openstack-ansible-upgrade-aio_metal-ubuntu-focal/85c4097/logs/openstack/ | 12:12 |
| noonedeadpunk | aio1-utility/stestr_results.html | 12:12 |
| noonedeadpunk | sheduler looks like it does it's job http://paste.openstack.org/show/803911/ | 12:13 |
| noonedeadpunk | and it even appears in compute log and being rescheduled, but never spawned for some reason | 12:16 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/85c40976164c4b1785aefe0c1a4ddab9/log/logs/host/nova-compute.service.journal-11-24-25.log.txt | 12:16 |
| openstackgerrit | Ebbex proposed openstack/openstack-ansible-os_tempest stable/rocky: Pin tempest-plugins to last tag supporting py2.7 https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/782990 | 12:17 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Fix condition when to create backends https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/782911 | 12:18 |
| openstackgerrit | Ebbex proposed openstack/openstack-ansible-lxc_hosts stable/rocky: Determine latest base image available https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/782734 | 12:18 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder stable/victoria: Fix condition when to create backends https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/782963 | 12:19 |
| *** shyamb has quit IRC | 12:20 | |
| *** gokhani has joined #openstack-ansible | 12:21 | |
| jrosser | noonedeadpunk: rabbit log looks kind of strange https://zuul.opendev.org/t/openstack/build/85c40976164c4b1785aefe0c1a4ddab9/log/logs/host/rabbitmq-server.service.journal-11-24-25.log.txt | 12:38 |
| noonedeadpunk | Well it looks like nokill systemd option is there | 12:39 |
| noonedeadpunk | or you mean `Logger - error: {removed_failing_handler,rabbit_log}`? | 12:40 |
| jrosser | maybe i expected to see some actual debug logging | 12:40 |
| jrosser | perhaps we dont enable any | 12:41 |
| jrosser | just from 10:37:57 to 11:13:42 (the exact point the tempest test bails out) theres nothing at all | 12:41 |
| jrosser | may just be co-incidence that 11:13:39 is the first point in the call logging from tempest | 12:43 |
| noonedeadpunk | I actually would expect to see the reason why instance was not spawned in compute log... | 12:45 |
| noonedeadpunk | instead of just the matter of fact that it hasn't | 12:45 |
| noonedeadpunk | doh | 12:48 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/85c40976164c4b1785aefe0c1a4ddab9/log/logs/host/nova-conductor.service.journal-11-24-25.log.txt#3215 | 12:48 |
| noonedeadpunk | `Connection failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate` | 12:49 |
| LowKey | noonedeadpunk: my gnocchi issue was fixed, i re-check one gnocchi container was down. 502 error was related to haproxy not able to reached one of the container, but this not shown on log, i just manually figure out.. | 12:57 |
| jonher | it should have shown in the haproxy log on the number of backends available at least, if not make sure the healthchecks are ok | 12:59 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Disable ssl for rabbitmq during upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/782996 | 13:03 |
| noonedeadpunk | yep, like `echo "show stat" | nc -U /var/run/haproxy.stat` | 13:05 |
| noonedeadpunk | but I'd say if haproxy can't reach one container, it should jsut go down. Which means that our check request is wrong | 13:05 |
| noonedeadpunk | and that would be great to find out what exactly and fix it | 13:06 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Disable ssl for rabbitmq during upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/782996 | 13:07 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump SHAs for master https://review.opendev.org/c/openstack/openstack-ansible/+/780434 | 13:08 |
| *** spatel_ has joined #openstack-ansible | 13:12 | |
| *** spatel_ is now known as spatel | 13:12 | |
| spatel | noonedeadpunk i have replaced SSD and not its showing 99% healthy | 13:13 |
| spatel | [root@ostack-infra-01 ~]# smartctl -A -d sat+cciss,0 /dev/sda | grep Wear_Leveling_Count | 13:13 |
| spatel | 177 Wear_Leveling_Count 0x0013 099 099 000 Pre-fail Always - 5 | 13:13 |
| noonedeadpunk | spatel: it's _showing_ you meant? | 13:18 |
| spatel | i meant i can see health, it was 001 and now its 099 | 13:18 |
| noonedeadpunk | yeah) | 13:19 |
| spatel | i am good and safe :) | 13:19 |
| noonedeadpunk | I'd say you replaced it just in time) | 13:19 |
| *** dwilde has joined #openstack-ansible | 13:19 | |
| spatel | I have just replaced single drive at present and i will replace next drive after 1 month so i will have big time window between two SSD lifetime | 13:20 |
| spatel | i don't want they go down sametime :) | 13:20 |
| *** gokhani61 has joined #openstack-ansible | 13:25 | |
| ebbex | https://review.opendev.org/c/openstack/openstack-ansible-tests/+/782799 Using cached setuptools-54.2.0.tar.gz ? should that be a problem on xenial? | 13:25 |
| *** gokhani has quit IRC | 13:26 | |
| *** jamesdenton has quit IRC | 13:26 | |
| *** jamesdenton has joined #openstack-ansible | 13:26 | |
| LowKey | thanks for the command noonedeadpunk | 13:28 |
| LowKey | yeah, i just simply destroy and create the container... instead of check the problem.. | 13:28 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Add trove tempest support https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/783002 | 13:33 |
| *** amalrajgenocidex has joined #openstack-ansible | 13:43 | |
| amalrajgenocidex | I deployed openstack-train with octavia-dashboard | 13:44 |
| amalrajgenocidex | but flavor and subnet is not loading in horizon | 13:45 |
| amalrajgenocidex | https://i.imgur.com/ZSB6ZIW.png | 13:45 |
| amalrajgenocidex | any idea? | 13:45 |
| noonedeadpunk | amalrajgenocidex: Does this user has right permissions? | 13:47 |
| noonedeadpunk | I mean octavia requires extra role assigned | 13:48 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Add trove tempest support https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/783002 | 13:48 |
| amalrajgenocidex | noonedeadpunk yes, I used openstack-ansible and made sure the user has all the permissions | 13:53 |
| amalrajgenocidex | indeed I can see the output in inspect element | 13:53 |
| amalrajgenocidex | https://i.imgur.com/YKl1oR1.png | 13:53 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Extend telemetry testing https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/783004 | 13:54 |
| noonedeadpunk | oh, right, it's admin user | 13:54 |
| noonedeadpunk | Oh, right | 13:55 |
| noonedeadpunk | I think it's because no flavors exist | 13:55 |
| noonedeadpunk | https://docs.openstack.org/octavia/train/admin/flavors.html#flavors | 13:56 |
| amalrajgenocidex | https://i.imgur.com/50QDqSb.png | 13:57 |
| johnsom | No flavors are required | 13:57 |
| amalrajgenocidex | I already tried creating multiple flavors | 13:57 |
| amalrajgenocidex | with both SINGLE and ACTIVE-STANDBY | 13:57 |
| noonedeadpunk | hm. I just recall some issue related to them with horizon plugin some time ago | 13:57 |
| jonher | might the red X and yellow warnings at the top of the inspect window be related? | 13:58 |
| noonedeadpunk | despite the were not required | 13:58 |
| johnsom | Are you sure you are using a recent version of dashboard? | 13:58 |
| amalrajgenocidex | johnsom yes, using latest version of stable/train octavia-dashboard | 13:59 |
| noonedeadpunk | btw I had it working for train for sure... | 13:59 |
| johnsom | Are there errors reported in the horizon log file? | 14:00 |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-haproxy_server master: Allow HAProxy stats to be pinned to one or more processes https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/783007 | 14:02 |
| noonedeadpunk | could think of some horizon specific policies, but it's an admin user... | 14:06 |
| amalrajgenocidex | yeah, don't think it's related to permissions because we can see the flavors in inspect element | 14:08 |
| noonedeadpunk | so rly, what are the errors in the console? | 14:08 |
| noonedeadpunk | browser console | 14:09 |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-haproxy_server master: Allow HAProxy stats to be pinned to one or more processes https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/783007 | 14:09 |
| amalrajgenocidex | https://i.imgur.com/JgBmcxC.png | 14:11 |
| amalrajgenocidex | noonedeadpunk it's related to barbican, | 14:11 |
| noonedeadpunk | and I assume you're able to create a LB with CLI? | 14:12 |
| noonedeadpunk | I found that bug I was talking about. https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/702464 | 14:14 |
| noonedeadpunk | we backported it back to train | 14:14 |
| noonedeadpunk | we backported fix of this bug | 14:15 |
| noonedeadpunk | so in case you're installing master octavia ui plugin, it ended up that way | 14:15 |
| openstackgerrit | Ebbex proposed openstack/openstack-ansible-tests stable/rocky: Update get-pip url https://review.opendev.org/c/openstack/openstack-ansible-tests/+/782799 | 14:22 |
| amalrajgenocidex | noonedeadpunk yes, backend is working | 14:25 |
| amalrajgenocidex | Let me go throgh the bug | 14:26 |
| amalrajgenocidex | I think it is not applicable here, we are already using horizon_git_track_branch | 14:30 |
| LowKey | noonedeadpunk: you have idea on this issue ? http://paste.openstack.org/show/NknFX8HH7GnwnGQkso5M/ , i'm currently create zone, but getting ouput "no_servers_configured" , something i miss up ? | 14:30 |
| amalrajgenocidex | I even tried using victoria branch for octavia-dashboard, and saw the same error , with availaibility zone errors also which is expected since octavia in train does not have availability zone | 14:31 |
| johnsom | Again, AZ do not need to be created, the dashboard will work with just the defaults. | 14:35 |
| noonedeadpunk | LowKey: how does your pools.yaml look like? | 14:37 |
| noonedeadpunk | amalrajgenocidex: ok, gotcha, was not sure how new your train deployment is. if it's there, it should be fine... | 14:38 |
| noonedeadpunk | so I'm kind of out of the good ideas | 14:38 |
| amalrajgenocidex | noonedeadpunk ok. No problem. Thanks for helping. Let me troubleshoot a bit more. | 14:40 |
| LowKey | noonedeadpunk: i dont have it, http://paste.openstack.org/show/803919/ , i need to manually create it ? | 14:40 |
| amalrajgenocidex | If didn't work, will have to fall back to cli method | 14:40 |
| LowKey | i can see it on /openstack/venvs/designate-22.1.0/lib/python3.8/site-packages/designate/tests/resources/pools_yaml/pools.yaml | 14:40 |
| noonedeadpunk | amalrajgenocidex: you can try setting `octavia_dashboard_git_install_branch: 4.0.1` as I have it running nicely in one of my train clouds atm | 14:43 |
| amalrajgenocidex | Ok. Thanks. Let me give it a try | 14:44 |
| noonedeadpunk | LowKey: https://opendev.org/openstack/openstack-ansible-os_designate/src/branch/master/defaults/main.yml#L104-L137 | 14:44 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Generate designate_pool_uuid dynamically https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/771841 | 14:44 |
| noonedeadpunk | You can also take a look at https://satishdotpatel.github.io//designate-integration-with-powerdns/ | 14:45 |
| noonedeadpunk | so actually designate usually does not face public endpoints and not used as a frontend. Instead you connect it to proper DNS cluster and designate manage it | 14:46 |
| LowKey | noonedeadpunk: ok thanks for guide | 14:50 |
| *** macz_ has joined #openstack-ansible | 14:58 | |
| *** juanoterocas has joined #openstack-ansible | 15:02 | |
| *** dwilde has quit IRC | 15:04 | |
| *** dwilde has joined #openstack-ansible | 15:06 | |
| *** jamesdenton has quit IRC | 15:08 | |
| *** jamesdenton has joined #openstack-ansible | 15:10 | |
| *** jamesdenton has quit IRC | 15:17 | |
| *** jamesdenton has joined #openstack-ansible | 15:18 | |
| *** yasemind has quit IRC | 15:37 | |
| *** yasemind has joined #openstack-ansible | 15:49 | |
| *** yasemind has quit IRC | 16:06 | |
| *** yasemind has joined #openstack-ansible | 16:12 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Use proper distro path for senlin and mistral https://review.opendev.org/c/openstack/openstack-ansible/+/777384 | 16:16 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Do not use imports inside included task https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/783047 | 16:17 |
| jrosser | \o/ 777384 merged, wow | 16:20 |
| *** mensis has quit IRC | 16:26 | |
| noonedeadpunk | I have no idea how https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/782796 has merged considering it should fail... | 16:27 |
| jrosser | oh dear! what error did we miss? | 16:42 |
| *** amalrajgenocidex has quit IRC | 16:44 | |
| openstackgerrit | Ebbex proposed openstack/openstack-ansible-tests stable/rocky: Update get-pip url https://review.opendev.org/c/openstack/openstack-ansible-tests/+/782799 | 16:47 |
| noonedeadpunk | jrosser: http://paste.openstack.org/show/803922/ | 16:48 |
| noonedeadpunk | 783047 fixing it... | 16:48 |
| noonedeadpunk | was deploying aio and catched that | 16:49 |
| *** rpittau is now known as rpittau|afk | 17:06 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:11 | |
| *** juanoterocas has quit IRC | 17:17 | |
| *** tobberydberg has quit IRC | 17:18 | |
| *** brad[] has quit IRC | 17:19 | |
| *** bverschueren has quit IRC | 17:20 | |
| *** bradm has quit IRC | 17:20 | |
| *** mugsie has quit IRC | 17:20 | |
| *** jhesketh has quit IRC | 17:20 | |
| *** arxcruz has quit IRC | 17:20 | |
| *** mcarden has quit IRC | 17:20 | |
| *** tacco has quit IRC | 17:20 | |
| *** pcaruana has quit IRC | 17:20 | |
| *** kukacz has quit IRC | 17:20 | |
| *** yasemind has quit IRC | 17:20 | |
| *** logan- has quit IRC | 17:20 | |
| *** jmccrory has quit IRC | 17:20 | |
| *** mugsie has joined #openstack-ansible | 17:20 | |
| *** tobberydberg has joined #openstack-ansible | 17:20 | |
| *** bverschueren has joined #openstack-ansible | 17:20 | |
| *** jmccrory has joined #openstack-ansible | 17:20 | |
| *** pcaruana has joined #openstack-ansible | 17:20 | |
| *** arxcruz has joined #openstack-ansible | 17:20 | |
| *** jhesketh has joined #openstack-ansible | 17:20 | |
| *** fridtjof[m] has quit IRC | 17:20 | |
| *** tinwood has quit IRC | 17:20 | |
| *** mrda has quit IRC | 17:20 | |
| *** akahat has quit IRC | 17:20 | |
| *** dpawlik has quit IRC | 17:20 | |
| *** tinwood has joined #openstack-ansible | 17:20 | |
| *** mrda has joined #openstack-ansible | 17:20 | |
| *** dpawlik has joined #openstack-ansible | 17:21 | |
| *** masterpe has quit IRC | 17:21 | |
| *** brad[] has joined #openstack-ansible | 17:21 | |
| *** gokhani61 has quit IRC | 17:21 | |
| *** manti has quit IRC | 17:21 | |
| *** logan- has joined #openstack-ansible | 17:21 | |
| *** kukacz has joined #openstack-ansible | 17:23 | |
| *** kukacz has quit IRC | 17:27 | |
| *** kukacz has joined #openstack-ansible | 17:27 | |
| *** akahat has joined #openstack-ansible | 17:30 | |
| noonedeadpunk | doh, and tempestconf is broken with unknown issue... | 17:32 |
| *** tinwood has quit IRC | 17:42 | |
| *** evrardjp has quit IRC | 17:42 | |
| *** cloudnull has quit IRC | 17:42 | |
| *** dmsimard has quit IRC | 17:42 | |
| *** janno_ has quit IRC | 17:42 | |
| *** dpawlik has quit IRC | 17:42 | |
| *** rodolof has quit IRC | 17:42 | |
| *** klamath_atx has quit IRC | 17:42 | |
| *** ebbex has quit IRC | 17:42 | |
| *** sshnaidm|afk has quit IRC | 17:42 | |
| *** LowKey has quit IRC | 17:42 | |
| *** Anticimex has quit IRC | 17:42 | |
| *** spy has quit IRC | 17:42 | |
| *** sc__ has quit IRC | 17:42 | |
| *** jroll has quit IRC | 17:42 | |
| *** gixx has quit IRC | 17:42 | |
| *** johanssone has quit IRC | 17:42 | |
| *** frickler has quit IRC | 17:42 | |
| *** admin0 has quit IRC | 17:42 | |
| *** openstackgerrit has quit IRC | 17:42 | |
| *** crazzy has quit IRC | 17:42 | |
| *** lemko has quit IRC | 17:42 | |
| *** chkumar|ruck has quit IRC | 17:42 | |
| *** gary_perkins has quit IRC | 17:42 | |
| *** jonher has quit IRC | 17:42 | |
| *** prometheanfire has quit IRC | 17:42 | |
| *** noonedeadpunk has quit IRC | 17:42 | |
| *** spotz has quit IRC | 17:42 | |
| *** akahat has quit IRC | 17:42 | |
| *** luksky has quit IRC | 17:42 | |
| *** owalsh has quit IRC | 17:42 | |
| *** zbr|rover has quit IRC | 17:42 | |
| *** poopcat has quit IRC | 17:42 | |
| *** trident has quit IRC | 17:42 | |
| *** partlycloudy has quit IRC | 17:42 | |
| *** corvus has quit IRC | 17:42 | |
| *** gouthamr has quit IRC | 17:42 | |
| *** gmann has quit IRC | 17:42 | |
| *** alanmeadows has quit IRC | 17:42 | |
| *** Open10K8S has quit IRC | 17:42 | |
| *** zbr|rover has joined #openstack-ansible | 17:42 | |
| *** Anticimex has joined #openstack-ansible | 17:42 | |
| *** corvus has joined #openstack-ansible | 17:42 | |
| *** owalsh has joined #openstack-ansible | 17:42 | |
| *** sshnaidm|afk has joined #openstack-ansible | 17:43 | |
| *** noonedeadpunk_ has joined #openstack-ansible | 17:43 | |
| *** jonher has joined #openstack-ansible | 17:43 | |
| *** spy has joined #openstack-ansible | 17:43 | |
| *** luksky has joined #openstack-ansible | 17:44 | |
| *** johanssone has joined #openstack-ansible | 17:44 | |
| *** evrardjp has joined #openstack-ansible | 17:44 | |
| *** rodolof has joined #openstack-ansible | 17:44 | |
| *** tinwood has joined #openstack-ansible | 17:44 | |
| *** trident has joined #openstack-ansible | 17:45 | |
| *** klamath_atx has joined #openstack-ansible | 17:45 | |
| *** gmann has joined #openstack-ansible | 17:45 | |
| *** janno has joined #openstack-ansible | 17:46 | |
| *** cyberpear has quit IRC | 17:46 | |
| *** partlycloudy has joined #openstack-ansible | 17:46 | |
| *** frickler_ has joined #openstack-ansible | 17:47 | |
| *** cyberpear has joined #openstack-ansible | 17:48 | |
| *** frickler_ has quit IRC | 17:49 | |
| *** akahat has joined #openstack-ansible | 17:50 | |
| *** cloudnull has joined #openstack-ansible | 17:51 | |
| *** jhesketh has quit IRC | 17:52 | |
| *** jhesketh has joined #openstack-ansible | 17:52 | |
| *** dwilde has quit IRC | 17:53 | |
| *** fridtjof[m] has joined #openstack-ansible | 17:54 | |
| *** fresta_ has quit IRC | 17:57 | |
| *** mubix has quit IRC | 17:57 | |
| *** mwhahaha has quit IRC | 17:57 | |
| *** johnsom has quit IRC | 17:57 | |
| *** NobodyCam has quit IRC | 17:57 | |
| *** melwitt has quit IRC | 17:57 | |
| *** CeeMac has quit IRC | 17:57 | |
| *** nicolasbock has quit IRC | 17:57 | |
| *** mnaser has quit IRC | 17:57 | |
| *** mwhahaha has joined #openstack-ansible | 17:57 | |
| *** CeeMac has joined #openstack-ansible | 17:57 | |
| *** fresta has joined #openstack-ansible | 17:57 | |
| *** NobodyCam has joined #openstack-ansible | 17:57 | |
| *** nicolasbock has joined #openstack-ansible | 17:57 | |
| *** melwitt has joined #openstack-ansible | 17:58 | |
| *** mubix has joined #openstack-ansible | 17:58 | |
| *** pcaruana has quit IRC | 17:58 | |
| *** jmccrory has quit IRC | 17:58 | |
| *** NewJorg has quit IRC | 17:58 | |
| *** MrClayPole has quit IRC | 17:58 | |
| *** dasp_ has quit IRC | 17:58 | |
| *** rpittau|afk has quit IRC | 17:58 | |
| *** jrosser has quit IRC | 17:58 | |
| *** snadge has quit IRC | 17:58 | |
| *** fyx has quit IRC | 17:58 | |
| *** hindret has quit IRC | 17:58 | |
| *** mmercer has quit IRC | 17:58 | |
| *** PrinzElvis has quit IRC | 17:58 | |
| *** sri_ has quit IRC | 17:58 | |
| *** jungleboyj has quit IRC | 17:58 | |
| *** fungi has quit IRC | 17:58 | |
| *** mnaser has joined #openstack-ansible | 17:58 | |
| *** johnsom has joined #openstack-ansible | 17:58 | |
| *** NewJorg has joined #openstack-ansible | 17:58 | |
| *** MrClayPole has joined #openstack-ansible | 17:58 | |
| *** dasp has joined #openstack-ansible | 17:58 | |
| *** fyx has joined #openstack-ansible | 17:58 | |
| *** hindret has joined #openstack-ansible | 17:58 | |
| *** PrinzElvis has joined #openstack-ansible | 17:58 | |
| *** sri_ has joined #openstack-ansible | 17:58 | |
| *** jrosser has joined #openstack-ansible | 17:58 | |
| *** rpittau|afk has joined #openstack-ansible | 17:58 | |
| *** jungleboyj has joined #openstack-ansible | 17:58 | |
| *** jmccrory has joined #openstack-ansible | 17:58 | |
| *** pcaruana has joined #openstack-ansible | 17:58 | |
| *** snadge has joined #openstack-ansible | 17:58 | |
| *** mmercer has joined #openstack-ansible | 17:58 | |
| *** openstack has joined #openstack-ansible | 18:05 | |
| *** ChanServ sets mode: +o openstack | 18:05 | |
| *** jungleboyj has quit IRC | 18:05 | |
| *** jrosser has quit IRC | 18:05 | |
| *** nicolasbock has quit IRC | 18:05 | |
| *** mwhahaha has quit IRC | 18:05 | |
| *** partlycloudy has quit IRC | 18:05 | |
| *** janno has quit IRC | 18:05 | |
| *** kukacz has quit IRC | 18:05 | |
| *** Daemoen has quit IRC | 18:05 | |
| *** persia has quit IRC | 18:05 | |
| *** persia has joined #openstack-ansible | 18:05 | |
| *** partlycloudy has joined #openstack-ansible | 18:05 | |
| *** Daemoen has joined #openstack-ansible | 18:05 | |
| *** trident has joined #openstack-ansible | 18:06 | |
| *** jungleboyj has joined #openstack-ansible | 18:06 | |
| *** cyberpear has quit IRC | 18:06 | |
| *** klamath_atx has quit IRC | 18:06 | |
| *** gmann has quit IRC | 18:06 | |
| *** Adri2000 has quit IRC | 18:06 | |
| *** Jeffrey4l has quit IRC | 18:06 | |
| *** janno has joined #openstack-ansible | 18:06 | |
| *** Jeffrey4l has joined #openstack-ansible | 18:06 | |
| *** gmann has joined #openstack-ansible | 18:07 | |
| *** klamath_atx has joined #openstack-ansible | 18:07 | |
| *** Adri2000 has joined #openstack-ansible | 18:07 | |
| *** Guest76033 has joined #openstack-ansible | 18:07 | |
| *** mwhahaha has joined #openstack-ansible | 18:07 | |
| *** cyberpear has joined #openstack-ansible | 18:07 | |
| *** jrosser has joined #openstack-ansible | 18:08 | |
| *** andrewbonney has quit IRC | 18:08 | |
| *** nicolasbock has joined #openstack-ansible | 18:08 | |
| *** cyberpear has quit IRC | 18:08 | |
| *** cyberpear has joined #openstack-ansible | 18:10 | |
| *** kukacz has joined #openstack-ansible | 18:10 | |
| *** akahat has joined #openstack-ansible | 18:10 | |
| *** mugsie has quit IRC | 18:16 | |
| *** macz_ has quit IRC | 18:16 | |
| *** jbadiapa has quit IRC | 18:16 | |
| *** waxfire has quit IRC | 18:16 | |
| *** priteau has quit IRC | 18:16 | |
| *** odyssey4me has quit IRC | 18:16 | |
| *** logan- has quit IRC | 18:16 | |
| *** mrda has quit IRC | 18:16 | |
| *** tobberydberg has quit IRC | 18:16 | |
| *** jamesdenton has quit IRC | 18:16 | |
| *** rh-jelabarre has quit IRC | 18:16 | |
| *** tosky has quit IRC | 18:16 | |
| *** Underknowledge has quit IRC | 18:16 | |
| *** mgariepy has quit IRC | 18:16 | |
| *** d34dh0r53 has quit IRC | 18:16 | |
| *** irclogbot_1 has quit IRC | 18:16 | |
| *** simondodsley has quit IRC | 18:16 | |
| *** gundalow has quit IRC | 18:16 | |
| *** Brace has quit IRC | 18:16 | |
| *** mugsie has joined #openstack-ansible | 18:16 | |
| *** Underknowledge has joined #openstack-ansible | 18:17 | |
| *** jamesdenton has joined #openstack-ansible | 18:17 | |
| *** macz_ has joined #openstack-ansible | 18:17 | |
| *** tosky has joined #openstack-ansible | 18:17 | |
| *** tobberydberg has joined #openstack-ansible | 18:18 | |
| *** logan- has joined #openstack-ansible | 18:18 | |
| *** odyssey4me has joined #openstack-ansible | 18:19 | |
| *** irclogbot_1 has joined #openstack-ansible | 18:20 | |
| *** rh-jelabarre has joined #openstack-ansible | 18:21 | |
| *** fridtjof[m] has quit IRC | 18:29 | |
| *** pcaruana has quit IRC | 18:31 | |
| *** dwilde has joined #openstack-ansible | 18:31 | |
| *** pcaruana has joined #openstack-ansible | 18:32 | |
| *** mgariepy has joined #openstack-ansible | 18:33 | |
| *** Guest84754 has joined #openstack-ansible | 18:46 | |
| *** fridtjof[m] has joined #openstack-ansible | 18:59 | |
| *** Guest84754 has quit IRC | 19:05 | |
| *** masterpe has joined #openstack-ansible | 19:07 | |
| *** manti has joined #openstack-ansible | 19:07 | |
| *** Guest84754 has joined #openstack-ansible | 19:09 | |
| *** spatel_ has joined #openstack-ansible | 19:15 | |
| *** spatel_ is now known as spatel | 19:15 | |
| spatel | noonedeadpunk_ do you know if i can change output formate of this command ansible compute_all -m shell -a "hostname" | 19:16 |
| *** jralbert has joined #openstack-ansible | 19:28 | |
| jralbert | jrosser: here's one of my compute nodes, running os-nova-install.yml limited to just that node, logging its attempt to build a venv for nova, and failing on cloning from opendev: http://paste.openstack.org/show/803931/ | 19:29 |
| mgariepy | jralbert, maybe stalled facts. | 19:34 |
| mgariepy | if the fact of the repo server are more than 24 hours old they might be ignored. | 19:35 |
| jrosser | yes, then it thinks there is no build server iirc | 19:36 |
| jralbert | So, if we include the repo containers in the limit, they'll get fresh facts and be used? | 19:36 |
| mgariepy | what's in your /root/.pip/pip.conf ? | 19:36 |
| jralbert | no such file | 19:37 |
| jrosser | which release is this? | 19:37 |
| mgariepy | train | 19:38 |
| jralbert | yep | 19:38 |
| jrosser | the thing that makes it use the repo server is this variable evaulating properly https://github.com/openstack/ansible-role-python_venv_build/blob/stable/train/tasks/python_venv_wheel_build.yml#L24 | 19:39 |
| mgariepy | ansible repo_all -m setup ? | 19:40 |
| mgariepy | to refresh the facts. | 19:40 |
| jrosser | this is what is making the fact cache timeout https://github.com/openstack/openstack-ansible/blob/master/scripts/openstack-ansible.rc#L42 | 19:42 |
| jralbert | but doesn't the task you linked to in the venv wheel build playbook specifically gather new facts for the build targets? | 19:44 |
| jralbert | because indeed it does reference the repo servers in our environment in that task: http://paste.openstack.org/show/803932/ | 19:45 |
| jralbert | But the block that follows it is nonetheless delegated to the compute node | 19:45 |
| spatel | jrosser noonedeadpunk_ i get sometime to put this together on my blog - https://satishdotpatel.github.io//openstack-ansible-inventory/ | 19:45 |
| jrosser | jralbert: maybe worth putting in debug tasks to print venv_build_host and work back to why it doesnt come out as expected | 19:49 |
| jrosser | a debug: printing venv_build_targets is probably a good move too | 19:50 |
| jralbert | Yes, I guess I'll have to | 19:51 |
| jrosser | i don't know if --limit adjusts the cached facts which are loaded | 19:53 |
| jrosser | it may well be that adding repo_all to your --limit helps | 19:54 |
| jrosser | becuase i suspect that this is where the "delegate to self" is coming from https://github.com/openstack/ansible-role-python_venv_build/blob/master/vars/main.yml#L64 | 19:54 |
| *** spatel has quit IRC | 19:58 | |
| *** zigo has joined #openstack-ansible | 20:03 | |
| *** spatel_ has joined #openstack-ansible | 20:05 | |
| *** spatel_ is now known as spatel | 20:05 | |
| *** spatel has quit IRC | 20:24 | |
| *** priteau has joined #openstack-ansible | 20:36 | |
| *** mcarden has joined #openstack-ansible | 21:23 | |
| *** mgariepy has quit IRC | 21:27 | |
| *** macz_ has quit IRC | 21:32 | |
| *** jamesdenton has quit IRC | 21:48 | |
| *** jamesdenton has joined #openstack-ansible | 21:49 | |
| *** openstackgerrit has joined #openstack-ansible | 21:53 | |
| openstackgerrit | Merged openstack/openstack-ansible-haproxy_server master: Allow HAProxy stats to be pinned to one or more processes https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/783007 | 21:53 |
| *** bradm has joined #openstack-ansible | 22:10 | |
| *** mgariepy has joined #openstack-ansible | 22:27 | |
| jralbert | I'm still totally stuck trying to get nova pulled down into the venvs on my compute nodes. I see that the repo containers have a current pull of the openstackgit repos; is there an easy way to tell the plays to target those copies instead of opendev? | 23:13 |
| *** rh-jelabarre has quit IRC | 23:18 | |
| fungi | i have a feeling your attempts may be correlated to the incidents where our git backends are falling over, if you're doing an ansible fireball asking 600 machines to clone nova at the same moment | 23:31 |
| fungi | we saw the problem start back up again between 22:25 and 22:30 utc | 23:32 |
| jralbert | So conveniently, all our attempts will appear to be coming from a single NAT'd outbound IP. If we're the ones what done it, you should be able to very clearly see that in your logs | 23:39 |
| jralbert | we are changing the OSA config to target github instead | 23:39 |
| fungi | well, i think the problem is the nature of the requests, 600 requests is a drop in the bucket, hardly moves the needle, but when the backend tries to load nova in memory that many times at once... | 23:41 |
| fungi | so that source address is doing a tiny fraction of the request count of, say, the typical search engine spiders which hammer the site continually | 23:42 |
| jralbert | do you have a candidate source address you think is to blame for those particular requests? | 23:42 |
| jralbert | if we can establish this is the cause of the issue, we may want to consider suggesting whether OSA by default should target the github mirrors instead of opendev | 23:43 |
| fungi | i don't which is why i'm not convinced that's it... when we disable the backend getting hit we sometimes see the problem move to a different backend, but when i try to correlate addresses which moved from the disabled backend to the next one which was observed to git hit, and then the next one after that's been disabled, the intersection of those sets is not conclusive | 23:44 |
| fungi | i was hoping to try to correlate by which times we saw the problem arise, but if you're not sure when you're running things, i suppose that won't help | 23:45 |
| jralbert | because the plays take so long to run, it's a bit tricky to pin down when we'd be making the requests, but I can try to get that if it'll be helpful | 23:46 |
| fungi | unfortunately the way the load balancer is implemented, it can't see the request details because it's not terminating ssl, but the backends don't know which client addresses they're talking to because the load balancer is the source address they see for each connection, so i can't correlate specific requests to specific clients | 23:47 |
| fungi | at least not without painstaking matching of source port numbers between the haproxy log and the apache log | 23:48 |
| fungi | jralbert: if you can confirm the playbook would have started asking servers to clone nova sometime around 22:25 or 22:30 utc, that might help | 23:49 |
| fungi | i'm mainly hoping to get some idea of what kinds of traffic trigger this scenario, so we can better design around them | 23:49 |
| jralbert | Well, I'm not sure if this will give you exactly what you want, but here's a collection of all the ansible log entries that correspond to the clone action during our upgrade, grouped by minute: http://paste.openstack.org/show/803936/ | 23:51 |
| fungi | thanks, i'll take a look | 23:51 |
| jralbert | Those timestamps should be UTC | 23:51 |
| fungi | also, i'll admit i'm not that familiar with how openstack-ansible is meant to operate, but if i were responsible for deploying updates of the same git repository to 600 servers i'd clone it once and then rsync it to them. i've seen what trying to repeatedly clone something from github is like and it's not pretty either | 23:52 |
| jralbert | Well, like a lot of openstack things, "it's complicated". OSA is intended to do exactly what you describe, and they've tried to limit the cost of builds - but, in big environments it gets hairy. | 23:53 |
| jralbert | so for us, running what I'd consider a medium-large openstack environment, the compute node plays take many many hours to run. In order to minimize disruption to our users, when we do a major version upgrade we do the control plane first, and then the compute nodes separately. | 23:54 |
| jralbert | This requires running the playbooks with limit flags, which can produce some unexpected results, as the tests are based on the assumption of an unlimited play run that targets all hosts | 23:55 |
| jralbert | So it looks like for us OSA is not understanding that it can use the repo servers it configures to host the git content, and is instead sourcing it from upstream every time | 23:56 |
| jralbert | Unfortunately for us, the first set of failures left us without a running nova-compute on many compute nodes, which produces a visible outage for users | 23:57 |
| fungi | yeah, that's definitely no good | 23:57 |
| jralbert | So we don't have a ton of time to do debug, we really need to get things running again in a hurry, and that's our focus right now | 23:58 |
| jralbert | I would be delighted to be involved in helping figure out a way to manage larger environments going forward, once we're out of the woods | 23:58 |
| fungi | well, i'm doing my best to keep our git servers up, i've not blocked anything, it's just too bad it can't space out the requests a bit | 23:58 |
| fungi | and yeah, due to our current load balancer and backend limitations, requests are persisted by source address so that multiple requests for a single git operation won't end up split between different backends and wind up looking for the wrong packfiles | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
