| *** sshnaidm|afk is now known as sshnaidm | 07:27 | |
| *** rpittau|afk is now known as rpittau | 07:30 | |
| opendevreview | Merged openstack/openstack-ansible-tests master: setup.cfg: Replace dashes with underscores https://review.opendev.org/c/openstack/openstack-ansible-tests/+/789761 | 09:28 |
|---|---|---|
| spatel | where is noonedeadpunk ?? | 13:03 |
| spatel | I have some patch which i like to get review so we can merge :) if someone else in core group please let me know | 13:30 |
| mgariepy | if the patches are not too big i can take a few minutes to check them out. | 13:34 |
| spatel | mgariepy its tiny change in OVN so yes impact is very low | 13:35 |
| spatel | 1. https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/802134 | 13:35 |
| spatel | 2. https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803060 | 13:36 |
| nurdie | Hey OSA! Trying to rebuild containers after a bunch of power outages wrecked my cluster. | 13:37 |
| nurdie | fatal: [infra1_utility_container-9f7e62cd]: FAILED! => {"changed": false, "msg": "Could not find the requested service systemd-networkd: host"} | 13:37 |
| nurdie | OSA 20.2.6 on CentOS 7 | 13:37 |
| nurdie | There's maybe an option somewhere on openstack_user_config? I've tried to trace this but I'm coming up short | 13:39 |
| spatel | nurdie run -vv and see where its not able to find it | 13:48 |
| spatel | mgariepy one more patch which is https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/802701 to get rid of centos-8 job | 13:49 |
| mgariepy | spatel, are the ovn ovsdb all clustered now ? | 13:49 |
| spatel | Yes | 13:49 |
| mgariepy | and everything is active-active ? | 13:50 |
| spatel | that patch is already merged, ovsdb cluster doesn't do active-active (its work like RabbitMQ) | 13:50 |
| spatel | one master and other slaves | 13:50 |
| spatel | leader node and follower (leader node only can write data and followers are just read-copy) | 13:51 |
| spatel | client will automatically detect leader and send write operation to leader only | 13:51 |
| spatel | when leader is dead, one of follower make itself leader after election | 13:52 |
| spatel | I believe we have addressed all OVN issues (except SSL related which i need to work) | 13:53 |
| mgariepy | rabbit is a-a irrc | 13:54 |
| spatel | no | 13:54 |
| spatel | In RabbitMQ only one node write and other just sync | 13:55 |
| spatel | As far as i know.. they are not multi-master | 13:55 |
| nurdie | spatel: it's not configuring it on any new containers. Additionally, the /etc/resolv.conf link to /var/run isn't getting setup either | 13:56 |
| nurdie | It's not configuring it on existing containers either, but that goes undectected because it's already configured there :p | 13:57 |
| spatel | mgariepy in rabbitMQ its little different like primary queue node, so one of node is primary for that queue and if you try to write data to secondary node (non-primary then that node forward your request to primary for write operation) | 14:00 |
| nurdie | Also, on train, OSA keeps configuring openstack-pike repo on servers >_< | 14:01 |
| nurdie | I just ran: openstack-ansible -vv playbooks/setup-hosts.yml -l infra1_repo_container-2aa293c7 | 14:03 |
| nurdie | It fails because no DNS config is on that container | 14:03 |
| nurdie | It fails here: TASK [openstack_hosts : Add requirement packages (repositories gpg keys packages, toolkits...)] | 14:03 |
| nurdie | https://pasteboard.co/Kebr9F4.png | 14:05 |
| nurdie | I can bandaid that but I want to fix the plays | 14:06 |
| spatel | You should keep centOS 7 version same don't change that otherwise repo server will create issue, i believe | 14:09 |
| mgariepy | spatel, i'm a bit confused by the ovs/ovn stuff. | 14:09 |
| spatel | if yum update is failed means look like you have issue somewhere in repo config | 14:10 |
| mgariepy | https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/setup_ovs_ovn.yml#L30 | 14:10 |
| mgariepy | wouldn't it need to be changed so we can get rid of the haproxy config? | 14:11 |
| spatel | ovs is just L2/L3 switch and OVN is SDN controller | 14:11 |
| spatel | yes we don't need haproxy stanza for OVN 6641 and 6642 | 14:13 |
| spatel | because we can point agent directly talk to member of cluster nodes | 14:13 |
| mgariepy | yeah this part was clear in my head. the issue i have is the `ovs-vsctl set open . external-ids:ovn-remote=tcp:{{ neutron_ovn_ip }}:6642` doesnt quite compute in my head haha | 14:13 |
| spatel | neutron_ovn_ip is haproxy vip which james initially configure before we implemented ovn cluster patch | 14:14 |
| mgariepy | can you then fix that part in subsequent patches ? | 14:15 |
| spatel | in my patch i am using list of cluster member nodes https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803060 | 14:16 |
| spatel | sure i can use same variable in that place | 14:16 |
| spatel | command: "ovs-vsctl set open . external-ids:ovn-remote={{ neutron_ovn_sb_connection }}" | 14:18 |
| spatel | something like that | 14:18 |
| mgariepy | or only on the self-ip address i'm not too sure. | 14:18 |
| mgariepy | but you are probably right about the connection string. | 14:19 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Use list of cluster member ipaddr for ovn ml2 agent to connect https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803060 | 14:20 |
| spatel | done check it our | 14:20 |
| spatel | it was a good catch | 14:21 |
| spatel | later i will create one more patch to remove ovn config from haproxy lb to just clean up | 14:22 |
| mgariepy | check my comment :D | 14:23 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Use list of cluster member ipaddr for ovn ml2 agent to connect https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803060 | 14:25 |
| spatel | done! | 14:25 |
| mgariepy | perfect | 14:25 |
| mgariepy | thanks | 14:25 |
| spatel | Thank for the review :) | 14:26 |
| nurdie | Should systemd-resolved be installed on _infrastructure_hosts as well as containers, or just containers? | 14:27 |
| spatel | mgariepy does Depends-On: will get merge itself or need separate +2 review for them? | 14:27 |
| mgariepy | they all need to be merged with +2 | 14:28 |
| spatel | nurdie i think so but i have never done that part, and never get any issue like that | 14:29 |
| nurdie | I've been running openstack for almost 3 years now and have never seen this either >_< | 14:30 |
| spatel | are you trying to upgrade something or changing OS.. ? | 14:30 |
| nurdie | Sequential power outages did a number. We even lost a whole compute node's RAID controller. Thankfully, we're full Ceph backend :) | 14:30 |
| nurdie | Nope | 14:30 |
| nurdie | Just recovering | 14:30 |
| spatel | again centos7 is old and anything is possible | 14:30 |
| nurdie | old usually means stable | 14:31 |
| spatel | not really, soon it will be EOL + sometime broken repo also cause issues | 14:31 |
| spatel | centos7 isn't part of CI-CD pipeline so hard to know.. what is broken and when. | 14:33 |
| nurdie | o.0 | 14:33 |
| nurdie | C7 doesn't EOL until 2024 | 14:33 |
| spatel | mgariepy do you know what is going on here - https://review.opendev.org/c/openstack/openstack-ansible/+/803041 | 14:33 |
| spatel | openstack-tox-docs https://zuul.opendev.org/t/openstack/build/589ec38bc3114399b722c3cb27f4fd75 : FAILURE in 5m 46s | 14:33 |
| nurdie | What should I upgrade to to be in CI? CentOS Stream? | 14:33 |
| spatel | OSA drop support of centos7 so its hard to debug.. (yes centos stream is new toy) | 14:34 |
| spatel | Recently i changed my deployment to use ubuntu (i am very happy) | 14:35 |
| spatel | centos is now puppet of redhat :) anytime they can change policy | 14:35 |
| nurdie | How did you do that? Just backup galera and do a full redeploy? | 14:35 |
| nurdie | I know....we aren't happy about it either | 14:35 |
| spatel | not centos7---> ubuntu migration (only new deployment will go to ubunut) | 14:36 |
| nurdie | ah. I won't have that luxury. It's a production cluster. I thought I had 2 more years to create a CERT cluster but I guess that needs to happen now lol | 14:37 |
| spatel | i am running centos7 cloud with 800 production vms and everyday i pray to god :) | 14:38 |
| nurdie | In theory, I should be able to easily backup galera and do full redeploy. All of my vms are on Ceph | 14:39 |
| spatel | i haven't think about how to upgrade this cluster but soon planning to migrate to ubuntu may be create parallel cluster and migrate each vm every days or lift and shift | 14:39 |
| spatel | nurdie if you can effort downtime then yes.. but i would say run test in lab | 14:39 |
| nurdie | i do not want to think about migrating 800 vms lol | 14:40 |
| nurdie | my heart goes out to you | 14:40 |
| spatel | In my infra nodes i replace SSD every 1 year to just make sure it won't die with SSD ( yes controller can die also but not thinking about that yet :) ) | 14:41 |
| nurdie | and, so far, you feel that ubuntu is far more stable than your centos stack? | 14:41 |
| spatel | Yes and very open.. not like centos keep hunting for packages and repos.. because they change them a lot.. | 14:42 |
| nurdie | man you should look in ceph. it's very easy to deploy and has been a total rock for me. you'll lose some IO perf, but you'll never lose prod data again | 14:42 |
| nurdie | "never" ;) | 14:42 |
| spatel | majority of cloud running on ubuntu if you look back survey again its personal choice so i don't want to force you :) | 14:43 |
| nurdie | hey i'm in this for stability. only started on centos because that's what the rest of prod runs on. i have zero qualms with adopting a deb fork | 14:43 |
| spatel | my application doesn't need any data storage, all i need memory, network and cpu for realtime processing data.. also i don't trust my network which can bring down my cloud or latency | 14:44 |
| spatel | for small deployment i can understand but we are running lots of vms | 14:45 |
| nurdie | You do have double the vms that I have.....but only double O:) | 14:46 |
| nurdie | My ceph cluster is all flash | 14:47 |
| nurdie | Works pretty alright | 14:47 |
| nurdie | But yeah, doesn't sound like that would be a worthwile investment for your ops | 14:47 |
| spatel | we do have ceph and it went down and brought down 200 vms since then we moved to local compute disk | 14:48 |
| spatel | ceph is good until its happy but it can create massive issue if not feeding carefully.. it required some skills which we don't have at present | 14:49 |
| nurdie | That sucks man sorry to hear that. It's never godo times when losing that much | 14:50 |
| mgariepy | spatel, not sure what's wrong with the docs on this. maybe spotz have some idea ? | 15:03 |
| spotz | reading | 15:13 |
| spotz | I haven't done a ton of Ceph and OpenStack to be honest. I can see if it'll run on my NUC but going through them is the only way II'll know | 15:15 |
| mgariepy | the error : /home/zuul/src/opendev.org/openstack/openstack-ansible/doc/source/admin/backup-restore.rst::rST localisation for language "id" not found | 15:23 |
| spatel | mgariepy something is changed recently somewhere broke that test | 15:26 |
| mgariepy | lol it seems like it's missing something from the id translation. | 15:27 |
| mgariepy | :/ | 15:27 |
| spatel | just curious why do we have that test only for this specific role and not others, can we set it to non-voting meantime | 15:28 |
| spotz | spatel: That would seem like a good idea if this is the only place it exists. Then we can evaluate it if we need it and possibly add it everywhere | 15:31 |
| spatel | mgariepy do you know how to set this to non-voting | 15:36 |
| mgariepy | in zuul.d/project-templates.yaml it can be set to non-voting. | 15:45 |
| spatel | is that file inside role? | 15:46 |
| spotz | Should be as each repo is tested separatelly | 15:47 |
| spatel | let me check | 15:48 |
| spatel | i can see it here - https://review.opendev.org/plugins/gitiles/openstack/openstack-ansible/+/refs/heads/master/zuul.d/project-templates.yaml | 15:50 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_nova master: Add dependency repo for centos-8-stream distro install https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/803368 | 15:55 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible master: set non-voting for broken tox-doc test https://review.opendev.org/c/openstack/openstack-ansible/+/803371 | 15:59 |
| spatel | mgariepy do we need to set +2 to merge this job? | 16:00 |
| mgariepy | spatel, you need to add : in the file ;p | 16:02 |
| mgariepy | - blah: then the voting: false | 16:02 |
| spatel | ? | 16:03 |
| spatel | https://review.opendev.org/c/openstack/openstack-ansible/+/803371/1/zuul.d/project-templates.yaml | 16:04 |
| spatel | did i miss something? | 16:04 |
| spatel | oh hold on | 16:04 |
| mgariepy | check my comment :D | 16:05 |
| spatel | +1 got it | 16:05 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible master: set non-voting for broken tox-doc test https://review.opendev.org/c/openstack/openstack-ansible/+/803371 | 16:06 |
| spatel | done :) | 16:06 |
| mgariepy | sphynx got updated recently. | 16:08 |
| spatel | that might broke something | 16:08 |
| spatel | also we should get rid of centos-8 jobs because anyway its EOL end of 2021 (it will reduce some load on build servers) | 16:10 |
| spatel | 5 more months to go | 16:10 |
| *** rpittau is now known as rpittau|afk | 16:13 | |
| spatel | mgariepy how to merge this stuff now - https://review.opendev.org/c/openstack/openstack-ansible/+/803371 | 19:06 |
| spatel | we need 2 +2 to get thing merge right? | 19:07 |
| mgariepy | wait a sec i'm didding on doc issue | 19:09 |
| jrosser | mgariepy: can you give this a push to help the infra folk out https://review.opendev.org/c/openstack/openstack-ansible-tests/+/803127 | 19:18 |
| mgariepy | jrosser, done. | 19:20 |
| mgariepy | the id translation seems to break the build. | 19:20 |
| jrosser | mgariepy: there was a post to the ML about that, I’ve just asked in the infra channel if anyone knows the fix | 19:22 |
| mgariepy | ok cool thanks jrosser | 19:26 |
| jrosser | I have a feeling hat the suggestion may end up being we are running the docs job when we don’t need to - i.e when we’re not modifying the contents of doc/ | 19:31 |
| mgariepy | it should build anyhow | 19:35 |
| mgariepy | if i remove the `id` translation the doc builds. | 19:35 |
| spatel | mgariepy how to do that part to fix broken test | 19:43 |
| spatel | jrosser could you give this guy push because its holding bunch of other patches https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/802134 | 19:45 |
| jrosser | spatel: done | 19:46 |
| jrosser | I’m on holidays this week, noonedeadpunk as well too I think | 19:46 |
| spatel | lucky guys! :) | 19:47 |
| spatel | jrosser i have fixed centos-8-stream disto install and here is the patch if you like to bump it up - https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/803368 | 19:48 |
| spatel | we should remove (non-voting) from openstack-ansible-deploy-aio_distro_metal-centos-8-stream | 19:49 |
| spatel | I am trying my best to keep centos-8-stream in good shape.. | 19:50 |
| jrosser | is os_nova really the place for a openvswitch repo? feels like networking imho? | 19:51 |
| spatel | nova rpms looking for openvswitch dependencies... | 19:52 |
| spatel | exact same repo we have in os_neutro role also but in setup-openstack.yml has nove playbook first and its dependency | 19:53 |
| opendevreview | Merged openstack/openstack-ansible-tests stable/train: Update Debian stable job https://review.opendev.org/c/openstack/openstack-ansible-tests/+/803127 | 20:00 |
| jrosser | spatel: feels very similar to this https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/vars/redhat-8.yml#L87 | 20:02 |
| spatel | Yes!! | 20:02 |
| spatel | we can move that here but only for distro method | 20:03 |
| jrosser | and then should we always get openvswitch from the same place for source/distro? | 20:03 |
| spatel | yes same place | 20:03 |
| spatel | for nova we don't need openvswitch but because of rpm dependency tree it asking to have that repo | 20:04 |
| spatel | we can move that repo in hosts if that is correct way to do | 20:05 |
| spatel | will looks more organized | 20:05 |
| spatel | jrosser can you give bump here so we can get rid of broken centos-8 build - https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/802701 | 20:05 |
| jrosser | done | 20:07 |
| spatel | cool! | 20:07 |
| spatel | i can try to move centos repo to here https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/vars/redhat-8.yml#L87 | 20:10 |
| spatel | add condition only for distro | 20:11 |
| spatel | in os_neutron we have condition only install when OVS deployment required so that is good | 20:11 |
| jrosser | is there a good reason to use a condition | 20:12 |
| jrosser | why don’t we always use that one? | 20:12 |
| spatel | Lets say if i want to install using source with LinuxBrige and don't want to use OVS then i don't need that repo | 20:14 |
| spatel | sure, we can use that repo also regardless of source or distro | 20:15 |
| spatel | not going to hurt | 20:15 |
| spatel | anyway i will push out that in hosts role and will see so we will have centralized place | 20:16 |
| spatel | I gotta go! you enjoy your holiday and have a safe vacation :) see you next week sometime | 20:17 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible stable/stein: Remove Debian Stable testing https://review.opendev.org/c/openstack/openstack-ansible/+/803404 | 23:04 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible stable/train: Remove Debian Stable testing https://review.opendev.org/c/openstack/openstack-ansible/+/803405 | 23:07 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
