| bline | that did work yankcrime, thanks | 00:33 |
|---|---|---|
| *** negronjl has quit IRC | 00:46 | |
| *** JangwonLee_ has quit IRC | 00:47 | |
| *** JangwonLee has joined #openstack-kolla | 00:48 | |
| *** negronjl has joined #openstack-kolla | 00:48 | |
| *** yingjun has joined #openstack-kolla | 01:05 | |
| *** yingjun has quit IRC | 01:05 | |
| *** zhanglong has joined #openstack-kolla | 01:32 | |
| *** zhanglong has quit IRC | 01:36 | |
| *** zhanglong has joined #openstack-kolla | 01:38 | |
| *** ricolin_ has joined #openstack-kolla | 02:22 | |
| *** ricolin_ has quit IRC | 02:31 | |
| *** vishalmanchanda has joined #openstack-kolla | 03:24 | |
| *** zhanglong has quit IRC | 03:28 | |
| *** zhanglong has joined #openstack-kolla | 03:28 | |
| *** evrardjp has quit IRC | 04:03 | |
| *** dave-mccowan has quit IRC | 04:06 | |
| *** evrardjp has joined #openstack-kolla | 04:10 | |
| *** zhanglong has quit IRC | 04:27 | |
| *** zhanglong has joined #openstack-kolla | 04:28 | |
| *** evrardjp has quit IRC | 04:36 | |
| *** evrardjp has joined #openstack-kolla | 04:36 | |
| *** Tony31 has joined #openstack-kolla | 04:50 | |
| *** ykarel|away is now known as ykarel | 04:50 | |
| Tony31 | morning :) | 04:50 |
| *** skramaja has joined #openstack-kolla | 05:02 | |
| *** JHTUI has joined #openstack-kolla | 05:18 | |
| JHTUI | Question to anyone out there. With Kolla Train, has anyone gotten Octavia to work in a multi-node cluster? I can get it to the point it spins up the amp instances for the LB, but the heartbeat fails with the compute nodes unable to communicate to the amp instances and the openstack loadbalancer command fails. How does one get around this? | 05:20 |
| *** ykarel is now known as ykarel|afk | 05:21 | |
| *** cah_link has joined #openstack-kolla | 05:25 | |
| JHTUI | it keeps failing with no route to host, despite having the API interface specified that works with other modules. security groups set up, opened up 5555 and the other ports, even did port 1-65535 both ways for giggles, attached a floating IP to the amp instances, the haproxy spins up fine on them, no errors in the containers. The issue isentirely | 05:25 |
| JHTUI | the controllers can't communicate. to the amp instances. Boggles me. | 05:25 |
| *** e0ne has joined #openstack-kolla | 05:38 | |
| *** ykarel|afk is now known as ykarel | 05:40 | |
| *** e0ne has quit IRC | 05:43 | |
| *** skramaja_ has joined #openstack-kolla | 06:01 | |
| *** skramaja_ has quit IRC | 06:02 | |
| *** skramaja_ has joined #openstack-kolla | 06:02 | |
| *** skramaja has quit IRC | 06:05 | |
| *** JHTUI has quit IRC | 06:18 | |
| *** e0ne has joined #openstack-kolla | 06:20 | |
| *** e0ne has quit IRC | 06:24 | |
| *** JHTUI has joined #openstack-kolla | 06:27 | |
| *** zhanglong has quit IRC | 06:30 | |
| *** zhanglong has joined #openstack-kolla | 06:34 | |
| Tony31 | hi JHTUI - no route to host means it may not be able to receive ARP response.. From the ansible host as well as all the nodes, can you ping all the nodes and the ansible host? | 06:35 |
| Tony31 | @JHTUI | 06:35 |
| *** Tony31 has quit IRC | 06:46 | |
| JHTUI | yes can ping the nodes fine, in fact once I attach floating ip's I can ssh into the containers | 06:49 |
| JHTUI | the octavia worker node just doesn't seem to be able to do the heartbeat/connectivity | 06:50 |
| JHTUI | set the amp network to the API network, verified the security works on the amps as adding port 22 allowed sshing into them, and when on the arp containers checking the logs on the containers showed the service was working and showed listening on the port. | 06:53 |
| JHTUI | essentially I'm running into the same issue the person who posted this is: https://ask.openstack.org/en/question/124991/octavia-worker-not-able-to-connect-to-amphora/ | 06:55 |
| JHTUI | tl;tr issue is: WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: ConnectTimeout: HTTPSConnectionPool(host='XXX.XXX.XXX.XXX', port=9443): Max retries exceeded with url: // | 06:58 |
| JHTUI | with the active/standby setup, it bounces between both AMP instances with the same message trying to establish connection until they both timeout and it bombs the Loadbalancer | 06:59 |
| JHTUI | been fighting this for several weeks. Rebuilt the kolla instance several dozen times trying to get this to work. | 07:01 |
| *** e0ne has joined #openstack-kolla | 07:01 | |
| JHTUI | I've even done silly things like trying to bind the amp interfaces to the public outfacing network, bind-linking new bridged networks onto the systems, still no dice | 07:04 |
| *** e0ne has quit IRC | 07:05 | |
| *** dougsz has joined #openstack-kolla | 07:12 | |
| *** bengates has joined #openstack-kolla | 07:14 | |
| *** bengates has quit IRC | 07:18 | |
| *** bengates has joined #openstack-kolla | 07:19 | |
| *** zhanglong has quit IRC | 07:24 | |
| *** abdysn has joined #openstack-kolla | 07:30 | |
| *** zhanglong has joined #openstack-kolla | 07:32 | |
| *** rpittau|afk is now known as rpittau | 07:34 | |
| *** arxcruz|off is now known as arxcruz | 07:35 | |
| *** jbadiapa has joined #openstack-kolla | 07:47 | |
| *** jbadiapa has quit IRC | 07:50 | |
| *** jbadiapa has joined #openstack-kolla | 07:51 | |
| *** srini_ has joined #openstack-kolla | 07:55 | |
| *** e0ne has joined #openstack-kolla | 07:56 | |
| *** e0ne has quit IRC | 08:01 | |
| *** zhanglong has quit IRC | 08:13 | |
| *** shyamb has joined #openstack-kolla | 08:15 | |
| *** srini_ has quit IRC | 08:15 | |
| hrw | morning | 08:19 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: move playbook files to tests/playbooks/ dir https://review.opendev.org/715872 | 08:19 |
| *** zhanglong has joined #openstack-kolla | 08:19 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible stable/train: requirements: use oslo.config <= 8.0.0 for py2 https://review.opendev.org/715873 | 08:24 |
| *** tonythomas has joined #openstack-kolla | 08:24 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: on aarch64 we only need to build images in upgrade jobs https://review.opendev.org/715875 | 08:28 |
| *** shyam89 has joined #openstack-kolla | 08:38 | |
| *** JHTUI has quit IRC | 08:40 | |
| *** shyamb has quit IRC | 08:41 | |
| *** skramaja_ has quit IRC | 08:41 | |
| *** skramaja_ has joined #openstack-kolla | 08:41 | |
| hrw | yoctozepto: ping https://review.opendev.org/#/c/715164 | 08:45 |
| patchbot | patch 715164 - kolla-ansible - CI: Always use upper-constraints - 1 patch set | 08:45 |
| *** shyam89 has quit IRC | 08:53 | |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe master: Install python-openstackclient using upper constraints https://review.opendev.org/715880 | 08:53 |
| *** shyam89 has joined #openstack-kolla | 08:54 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: Always use upper-constraints https://review.opendev.org/715164 | 08:54 |
| *** Tony31 has joined #openstack-kolla | 08:58 | |
| Tony31 | hi guys - hope you are keeping safe and well | 08:58 |
| *** kevko_ has joined #openstack-kolla | 08:59 | |
| *** kevko has joined #openstack-kolla | 08:59 | |
| Tony31 | @mgoddard how are you doing? | 08:59 |
| *** k_mouza has joined #openstack-kolla | 08:59 | |
| Tony31 | I'm running into this bug that it looks like you've pushed a fix to https://github.com/openstack/kolla-ansible/commit/93a4dcc1b97b1d7bbd3b91cc9c4bd4617ec6f9ba | 08:59 |
| openstackgerrit | Pierre Riteau proposed openstack/kolla-ansible master: Support setting Kafka storage volume https://review.opendev.org/715307 | 09:00 |
| Tony31 | having some troubles pulling down the fix, though. Do I do that with `kayobe control host upgrade` ? | 09:01 |
| hrw | Tony31: safe, well and bored | 09:02 |
| Tony31 | good to see you hrw | 09:04 |
| mnasiadka | morning | 09:04 |
| Tony31 | morning mnasiadka | 09:04 |
| *** cah_link1 has joined #openstack-kolla | 09:20 | |
| *** cah_link has quit IRC | 09:21 | |
| *** cah_link1 is now known as cah_link | 09:21 | |
| *** shyam89 has quit IRC | 09:30 | |
| *** ykarel is now known as ykarel|lunch | 09:32 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: move playbook files to tests/playbooks/ dir https://review.opendev.org/715872 | 09:32 |
| mgoddard | hi Tony31, that fix should be in the stable branches. stable/<release> | 09:37 |
| Tony31 | Hi @mgoddard - I am not sure how to pull it down. I am running `kayobe control host upgrade` but I get an error that I've modified some files. So I Then renamed `/etc/kolla` to `/etc/kolla.old` but I get the same error so I'm a bit confused now | 09:38 |
| Tony31 | "fatal: [localhost]: FAILED! => {"before": "9770e596dd0a18d733f8682f32d7cf6b328739c5", "changed": false, "msg": "Local modifications exist in repository (force=no)."} | 09:39 |
| Tony31 | " | 09:39 |
| Tony31 | can I force=yes ? | 09:39 |
| hrw | mgoddard, mnasiadka: can you take a look at some patches? I would prefer to know should they be worked on or dropped. | 09:40 |
| mnasiadka | hrw: #define some :) | 09:40 |
| hrw | https://review.opendev.org/#/c/713135/ https://review.opendev.org/#/c/713134/ https://review.opendev.org/#/c/715164/ https://review.opendev.org/715202 | 09:40 |
| patchbot | patch 713135 - kolla - CI: Avoid tox - 33 patch sets | 09:40 |
| patchbot | patch 713134 - kolla-ansible - CI: Avoid tox (and clean up gate setup) - 10 patch sets | 09:40 |
| patchbot | patch 715164 - kolla-ansible - CI: Always use upper-constraints - 2 patch sets | 09:40 |
| patchbot | patch 715202 - kolla - CI: Ansiblify setup of builders - 8 patch sets | 09:40 |
| mgoddard | hrw: I'm out until Thursday this week | 09:40 |
| hrw | https://review.opendev.org/715166 just waits for other patches | 09:40 |
| patchbot | patch 715166 - kolla - Revert "openstack-base: pin setuptools < 46 to get... - 1 patch set | 09:40 |
| hrw | mgoddard: ok | 09:40 |
| openstackgerrit | Will Szumski proposed openstack/kayobe master: Switch to stackhpc fork of resmo.ntp https://review.opendev.org/715893 | 09:42 |
| Tony31 | anyone here know how to update kolla ansible from kayobe ? | 09:49 |
| Tony31 | maybe it's not kolla-ansible I need to update? | 09:50 |
| Tony31 | trying to pull the fix for this bug https://github.com/openstack/kolla-ansible/commit/93a4dcc1b97b1d7bbd3b91cc9c4bd4617ec6f9ba | 09:50 |
| yankcrime | bline: awesome | 09:52 |
| Tony31 | @mgoddard - when using kayobe, how does the ansible files get installed to /etc/kolla ? | 09:52 |
| dougsz | Tony31: $kayobe control host bootstrap | 09:53 |
| Tony31 | @dougsz I can try it - thanks | 09:53 |
| dougsz | Will install the KA as set in you kolla.yml file | 09:53 |
| Tony31 | @dougsz - thanks mate. that part is default - having a read now | 09:54 |
| dougsz | np - kolla_ansible_source_url and kolla_ansible_source_version are probably the vars of interest to you | 09:55 |
| *** zhanglong has quit IRC | 09:56 | |
| *** cah_link1 has joined #openstack-kolla | 10:00 | |
| *** cah_link has quit IRC | 10:01 | |
| *** cah_link1 is now known as cah_link | 10:01 | |
| *** zhanglong has joined #openstack-kolla | 10:02 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: on aarch64 we only need to build images in upgrade jobs https://review.opendev.org/715875 | 10:05 |
| *** abdysn has quit IRC | 10:12 | |
| *** ykarel|lunch is now known as ykarel | 10:15 | |
| dougsz | Any thoughts on whether to support multiple instances of MariaDB on the same host? | 10:21 |
| dougsz | https://review.opendev.org/#/c/619756 | 10:21 |
| patchbot | patch 619756 - kolla-ansible - Minimal support for deploying multiple instances o... - 19 patch sets | 10:21 |
| dougsz | ^ part of that patch | 10:21 |
| *** dciabrin has joined #openstack-kolla | 10:21 | |
| dougsz | I.e. for cells with shared cell controller group, should all cells use the same MariaDB cluster, or each have their own? | 10:22 |
| dougsz | Having their own makes the cells less coupled, and cgroups could be used to limit resources used for each MariaDB cluster | 10:23 |
| dougsz | Sharing a DB cluster on the other hand is simpler | 10:23 |
| *** whoami-rajat is now known as whoami-rajat|lun | 10:25 | |
| *** whoami-rajat|lun is now known as whoami-rajat | 10:25 | |
| *** cah_link has quit IRC | 10:27 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: WIP: Remove support for CentOS 7 https://review.opendev.org/692450 | 10:33 |
| hrw | gerrit complained about merge conflicts ;d | 10:33 |
| *** Wellie has joined #openstack-kolla | 10:36 | |
| Wellie | Hello | 10:36 |
| *** skramaja_ has quit IRC | 10:39 | |
| *** skramaja_ has joined #openstack-kolla | 10:39 | |
| *** shyamb has joined #openstack-kolla | 10:39 | |
| kevko | hi, is this fixed by other way or still a bug ? https://bugs.launchpad.net/kolla/+bug/1844574 | 10:44 |
| openstack | Launchpad bug 1844574 in kolla "td-agent user does not have permission to create /var/run/fluentd" [Medium,Expired] | 10:44 |
| *** zhanglong has quit IRC | 10:45 | |
| *** dmellado has quit IRC | 10:50 | |
| *** rpittau is now known as rpittau|bbl | 10:56 | |
| *** e0ne has joined #openstack-kolla | 11:00 | |
| *** cah_link has joined #openstack-kolla | 11:02 | |
| *** e0ne has quit IRC | 11:05 | |
| *** e0ne has joined #openstack-kolla | 11:05 | |
| *** JangwonLee_ has joined #openstack-kolla | 11:08 | |
| *** JangwonLee has quit IRC | 11:11 | |
| *** JangwonLee__ has joined #openstack-kolla | 11:16 | |
| *** JangwonLee_ has quit IRC | 11:19 | |
| *** gfidente has quit IRC | 11:21 | |
| hrw | meh. | 11:25 |
| hrw | docker_image_tag is set and not used | 11:25 |
| hrw | at least looks that way | 11:26 |
| hrw | https://416277a5687e0626ea6a-8dbb68c995c97f9fb5b4258bb26a4c49.ssl.cf5.rackcdn.com/715875/2/check-arm64/kolla-ansible-debian-source-aarch64/7b56736/job-output.json lists docker_image_tag == master-aarch64 like it is set in run.yml | 11:27 |
| *** JangwonLee_ has joined #openstack-kolla | 11:27 | |
| hrw | https://416277a5687e0626ea6a-8dbb68c995c97f9fb5b4258bb26a4c49.ssl.cf5.rackcdn.com/715875/2/check-arm64/kolla-ansible-debian-source-aarch64/7b56736/primary/logs/ansible/deploy uses "kolla/debian-source-fluentd:master" image | 11:27 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: on aarch64 we only need to build images in upgrade jobs https://review.opendev.org/715875 | 11:29 |
| hrw | hope it fixes | 11:30 |
| *** JangwonLee__ has quit IRC | 11:30 | |
| openstackgerrit | Piotr Rabiega proposed openstack/kolla master: add dpdk_telemetry and logparser https://review.opendev.org/715923 | 11:40 |
| *** JangwonLee__ has joined #openstack-kolla | 11:55 | |
| *** JangwonLee_ has quit IRC | 11:59 | |
| *** skramaja_ has quit IRC | 12:03 | |
| *** Fl1nt has joined #openstack-kolla | 12:03 | |
| Fl1nt | Good afternoon gentlemen ! | 12:03 |
| *** e0ne has quit IRC | 12:04 | |
| *** e0ne has joined #openstack-kolla | 12:04 | |
| *** dmellado has joined #openstack-kolla | 12:06 | |
| *** shyamb has quit IRC | 12:16 | |
| mgoddard | Fl1nt: and people of any gender! | 12:17 |
| Fl1nt | that's an implied expression so yes. | 12:20 |
| *** rpittau|bbl is now known as rpittau | 12:46 | |
| openstackgerrit | Doug Szumski proposed openstack/kayobe stable/train: Fix loading of Monasca Grafana dashboards https://review.opendev.org/715934 | 12:55 |
| *** oyrogerg has quit IRC | 12:56 | |
| openstackgerrit | Doug Szumski proposed openstack/kayobe master: Fix loading of Monasca Grafana dashboards https://review.opendev.org/715638 | 12:57 |
| *** oyrogerg has joined #openstack-kolla | 12:58 | |
| *** dave-mccowan has joined #openstack-kolla | 12:59 | |
| *** Luzi has joined #openstack-kolla | 13:02 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: on aarch64 we only need to build images in upgrade jobs https://review.opendev.org/715875 | 13:04 |
| *** gfidente has joined #openstack-kolla | 13:07 | |
| yoctozepto | dougsz: would it be hard to deliver the more flexbile approach? would delivering simplified one make it hard to introduce the flexible one? | 13:10 |
| *** irclogbot_3 has quit IRC | 13:13 | |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe master: Fix loading of Monasca Grafana dashboards https://review.opendev.org/715638 | 13:14 |
| hrw | uf. this version looks like working one | 13:14 |
| *** irclogbot_3 has joined #openstack-kolla | 13:18 | |
| dougsz | yoctozepto: I will rebase the patch and report back, thanks | 13:19 |
| *** TrevorV has joined #openstack-kolla | 13:24 | |
| *** Luzi has quit IRC | 13:26 | |
| *** cah_link1 has joined #openstack-kolla | 13:29 | |
| *** jimcrowleyibm has joined #openstack-kolla | 13:30 | |
| *** dking_desktop has joined #openstack-kolla | 13:30 | |
| *** cah_link has quit IRC | 13:31 | |
| *** cah_link1 is now known as cah_link | 13:31 | |
| *** jimcrowleyibm has quit IRC | 13:32 | |
| *** jimcrowleyibm has joined #openstack-kolla | 13:33 | |
| *** noxoid has joined #openstack-kolla | 13:40 | |
| *** dmellado has quit IRC | 14:04 | |
| *** dmellado has joined #openstack-kolla | 14:05 | |
| *** e0ne has quit IRC | 14:15 | |
| *** noxoid has quit IRC | 14:23 | |
| *** ykarel is now known as ykarel|away | 14:25 | |
| hrw | openstack is full of crap and lack of useful stuff | 14:44 |
| hrw | fg;e3hgy7te398ty2# | 14:44 |
| hrw | why cinder lacks 'image create --from-url https://somewhere' is unknown to me | 14:45 |
| *** klippo has left #openstack-kolla | 14:46 | |
| *** noxoid has joined #openstack-kolla | 14:59 | |
| *** yoctozepto has quit IRC | 15:07 | |
| *** yoctozepto has joined #openstack-kolla | 15:08 | |
| *** osmanlicilegi has quit IRC | 15:08 | |
| *** ab-a has quit IRC | 15:08 | |
| *** arxcruz has quit IRC | 15:09 | |
| *** obre has quit IRC | 15:09 | |
| *** osmanlicilegi has joined #openstack-kolla | 15:10 | |
| *** pescobar has quit IRC | 15:10 | |
| *** EmilienM has quit IRC | 15:10 | |
| *** arxcruz has joined #openstack-kolla | 15:10 | |
| *** ab-a has joined #openstack-kolla | 15:11 | |
| *** EmilienM has joined #openstack-kolla | 15:11 | |
| *** JHTUI has joined #openstack-kolla | 15:13 | |
| *** pescobar has joined #openstack-kolla | 15:20 | |
| *** ab-a has quit IRC | 15:26 | |
| mnasiadka | hrw: cinder and image? | 15:27 |
| *** ab-a has joined #openstack-kolla | 15:30 | |
| *** dmellado has quit IRC | 15:32 | |
| hrw | whatever | 15:38 |
| hrw | openstack cli sucks terribly | 15:38 |
| hrw | python-SERVICEclient -> python-openstackclient migration started in ocata (or earlier) and is still in progress | 15:39 |
| *** andreykurilin has quit IRC | 15:39 | |
| hrw | you use one command to list, other to use etc | 15:39 |
| *** obre has joined #openstack-kolla | 15:39 | |
| *** dmellado has joined #openstack-kolla | 15:39 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Revert "openstack-base: pin setuptools < 46 to get horizon working" https://review.opendev.org/715166 | 15:44 |
| *** ab-a has quit IRC | 15:45 | |
| *** r3ap3r has quit IRC | 15:45 | |
| *** cz3 has quit IRC | 15:45 | |
| *** andrein has quit IRC | 15:45 | |
| *** niedbalski has quit IRC | 15:45 | |
| *** rpittau has quit IRC | 15:45 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Revert "openstack-base: pin setuptools < 46 to get horizon working" https://review.opendev.org/713681 | 15:45 |
| mnasiadka | hrw: yeah well, the usability part of OpenStack always sucked big time, there are some really useful API calls that are not covered in any client :) | 15:45 |
| *** ab-a has joined #openstack-kolla | 15:45 | |
| *** r3ap3r has joined #openstack-kolla | 15:45 | |
| *** cz3 has joined #openstack-kolla | 15:45 | |
| *** andrein has joined #openstack-kolla | 15:45 | |
| *** niedbalski has joined #openstack-kolla | 15:45 | |
| *** rpittau has joined #openstack-kolla | 15:45 | |
| mnasiadka | hrw: so you end up running curl :) | 15:45 |
| hrw | mnasiadka: Debian -300Mbps-> PL -20Mbps-> US | 15:46 |
| *** andreykurilin has joined #openstack-kolla | 15:46 | |
| hrw | looks like perfect waste of time and bandwidth | 15:46 |
| kplant | <hrw> why cinder lacks 'image create --from-url https://somewhere' is unknown to me | 15:48 |
| *** jovial[m] has quit IRC | 15:48 | |
| kplant | glance _not_ having that slows down my builds SO much | 15:48 |
| hrw | cinder/glance/whateverstoresimageinallinoneormultinode | 15:49 |
| kplant | i empathize | 15:49 |
| mnasiadka | kplant: propose a patch? :) | 15:50 |
| hrw | like I wrote on twitter... | 15:50 |
| hrw | It was hard to write so should be hard to use and hard to understand. | 15:50 |
| hrw | Feels like motto of #OpenStack projects... | 15:50 |
| mnasiadka | hrw: well, OpenStack has a history of not caring for how to use it - and now with a lot of companies focus elsewhere - it's not easy to find people to work on the CLI | 15:53 |
| kplant | perhaps an entire release could be tanked and converted into a technical debt release? | 15:55 |
| Fl1nt | Well, that's the issue with every single community would them be open sources or gaming, you've a hype that bring many, then elits decide that you don't need what you're looking for, then they finish to be alone in there own kingdom but it's an empty castle | 15:55 |
| JHTUI | Anyone here get octavia working on kolla in a multi-node setup? | 15:56 |
| Fl1nt | yep | 15:56 |
| Fl1nt | stein release | 15:57 |
| Fl1nt | but yep | 15:57 |
| noxoid | same, octavia on stein works for me | 15:57 |
| *** hjensas has quit IRC | 15:57 | |
| JHTUI | I'm running into an issue where the octavia-workers can't talk to the amphora instances for the heartbeat/connectivity check. Any idea how to get past that? | 15:57 |
| hrw | mnasiadka: yeah | 15:57 |
| Fl1nt | @JHTUI, look at your CA Cert, then at your management network. | 15:58 |
| hrw | https://review.opendev.org/715875 looks like success. fails on aarch64 still but that's unrelated faulure after deploying/run | 15:58 |
| patchbot | patch 715875 - kolla-ansible - CI: on aarch64 we only need to build images in upg... - 4 patch sets | 15:58 |
| JHTUI | it gets past the CA cert validation. Error isoctavia.amphorae.drivers.haproxy.rest_api.driver Could not connect to instance Retrying (over and over) with connect timeout | 16:00 |
| JHTUI | to port 9443 | 16:00 |
| Fl1nt | ok, you can have a look at your amphorae instance, within the logs and then throw a tcpdump to see if you can get the packets actually reach your management interface. | 16:01 |
| * hrw out | 16:01 | |
| JHTUI | I've verified that the octavia worker node can't send/receive packets to the amp instances, and the question is I don't know why. Per the kolla documentation, I have the bonded network as the API interface, and set up networking and security as they expected. For giggles I even oopened up ports 1:65535 to remove all possibilities | 16:03 |
| Fl1nt | @JHTUI, did you created the mgmt-net ? | 16:04 |
| JHTUI | yes. | 16:04 |
| Fl1nt | and you instance get an IP from it within the namespace? | 16:05 |
| JHTUI | with the uuid assigned as the octavia network in the globals.yml | 16:05 |
| JHTUI | yes both amph instances get IP's | 16:05 |
| JHTUI | I've tried silly things like attempting to bind the LB's to a public facing networks as well | 16:06 |
| *** bengates has quit IRC | 16:07 | |
| Fl1nt | put yourself on this network gateway and do a tcpdump, you should be able to see packets. | 16:07 |
| Fl1nt | Can't remember exactly but each time that I get my amphorae that couldn't be reached by the worker it was either a routing issue, a namespace issue or a certificate issue. | 16:08 |
| JHTUI | this one looks like a routing issue. So the question is if I set all the globals correctly yet it still can't route, what's missing? | 16:09 |
| Fl1nt | who manage your networks within your company/platform? | 16:10 |
| Fl1nt | it may be a missing vlan distribution | 16:10 |
| Fl1nt | or a missing route declaration | 16:10 |
| JHTUI | we do, and I already verified the API bond interface (bond2) can reach all the hardware in the cluster | 16:10 |
| Fl1nt | at TOR or Core swithc | 16:10 |
| Fl1nt | which kind of equipment is the gateway of your mgmt-net ? physical switch? OpenVswitch? | 16:11 |
| Fl1nt | which type of network did you used as mgmt-net ? | 16:12 |
| *** e0ne has joined #openstack-kolla | 16:12 | |
| Fl1nt | vxlan? | 16:12 |
| JHTUI | vxlan | 16:12 |
| Fl1nt | did you tried to create a VM on this mgmt-net and to send it packets from your control plan ? | 16:12 |
| *** rpittau is now known as rpittau|afk | 16:12 | |
| JHTUI | yea, that part works. | 16:12 |
| JHTUI | this is probably user error on my end, but I can't see where I messed this up | 16:13 |
| Fl1nt | wait a second,when placed within the amphorae you see packets incoming? | 16:13 |
| JHTUI | I've rebuilt it so many times I'm seeing octavia in my dreams | 16:13 |
| JHTUI | I'm seeing packets going over the bridge, but not being received by amphorae, that's why I'm head scratching | 16:14 |
| JHTUI | it's *got* to be a routing issue, but I don't know where | 16:14 |
| JHTUI | been trying to find examples of working octavia setups so I could compare to what I have to see what I screwed up on, but the documentation on howtos are woefully lacking | 16:15 |
| Fl1nt | There is something that I can't figure out, you told me "that part works" when asked to put a VM within the MGMT-NET and send traffic to it from the control plan, then your amphorae instances don't? | 16:15 |
| Fl1nt | Did you created the appropriate secgroup? | 16:15 |
| JHTUI | yes, secgroup has 5555, 22, and 9443 opened. Getting frustrated I also opened all of udp and tcp fro 1:65535 to see if it was blocking on a non-registered port as well | 16:16 |
| Fl1nt | when you're trying to capture the packets, do you put yourself within the amphorae namespace? | 16:17 |
| JHTUI | I actually dropped a floating ip to theamp instances, logged in and ran tcpdump on them | 16:18 |
| Fl1nt | that won't work, you need to capture the traffic from within the namespace. | 16:18 |
| JHTUI | hum ok. | 16:18 |
| JHTUI | I'll check that out. Do you have an example octavia setup I can compare against on a known working install? Not having one I'm finding it hard to compare | 16:19 |
| JHTUI | been at this for a few weeks so I'm frustrated | 16:19 |
| JHTUI | doesn't help that it takes about 45 minutes for the LB to timeout and fail | 16:20 |
| *** Tony31 has quit IRC | 16:21 | |
| *** kevinz has quit IRC | 16:22 | |
| noxoid | JHTUI, sec, i can provide you an example | 16:27 |
| noxoid | JHTUI, http://paste.openstack.org/show/3E65oM8YPx8dFHhimW4i/ | 16:30 |
| noxoid | only thing i intentionally omitted was the config dir with the octavia certs | 16:31 |
| noxoid | let me know if theres anything else that may be useful | 16:31 |
| JHTUI | did you have to add specific interfaces on the compute/controllers to bridge the connections outside the kolla install for this? | 16:34 |
| noxoid | on the compute side neutron handles the configuration (via openvswitch). i manually set up the bond1.1002 interface on the controller via kickstart. vlan1002 is trunked to both the compute nodes and controller nodes at the top-of-rack | 16:35 |
| *** evrardjp has quit IRC | 16:36 | |
| *** evrardjp has joined #openstack-kolla | 16:36 | |
| *** kevko_ has quit IRC | 16:37 | |
| *** kevko has quit IRC | 16:37 | |
| *** hjensas has joined #openstack-kolla | 16:39 | |
| JHTUI | noxoid, understood on the ca certs, could you let me know what the subnet is for your management network? | 16:39 |
| noxoid | my octavia management network? | 16:41 |
| JHTUI | yea, is it in the same subnet as the bond1 trunk? | 16:42 |
| noxoid | its in the paste, 10.102.0.0/21 | 16:42 |
| JHTUI | ahah, ok, I think that's what I screwed up on. I had my lb management subnet larger than the API bond interface | 16:42 |
| noxoid | if you're talking about api_interface, i have mine completely separate | 16:43 |
| noxoid | api_interface: "bond1.1001" | 16:43 |
| noxoid | octavia_network_interface: "bond1.1002" | 16:43 |
| *** bengates has joined #openstack-kolla | 16:43 | |
| JHTUI | hum. ok. so your management interface is sitting on the new trunk you then set up on the controllers | 16:44 |
| noxoid | you dont have to use separate networks if you're ok with amphora vms accessing your api_interface network | 16:44 |
| noxoid | which id recommend against | 16:44 |
| noxoid | for a production install anyways | 16:44 |
| JHTUI | agreed. Right now we're just trying a POC so at this point I just want it up, I'm a week behind and my manager is getting twitchy :) | 16:45 |
| generalfuzz | Welcoming any additional reviews for our initial backend tls implementation for Keystone service: https://review.opendev.org/#/c/713986/ | 16:47 |
| patchbot | patch 713986 - kolla-ansible - Add support for encrypting backend HAProxy traffic... - 5 patch sets | 16:47 |
| JHTUI | surprised that kolla's module for octavia doesn't do this for you noxoid. It's all ansible driven so theoretically it wouldn't be hard. | 16:47 |
| noxoid | its possible but i like to think kolla doesnt do it as octavia deployments can be fairly site-specific | 16:48 |
| johnsom | I agree, it would be great to have kolla handle this | 16:48 |
| JHTUI | yea, but even still it should be easy enough to make a fact sheet for deploy. it's the entire purpose of ansible. | 16:48 |
| johnsom | There seem to be a lot of questions about this and OSA and tripleo both set this up for the user. | 16:49 |
| noxoid | yea this isnt the first person ive tried to help with octavia | 16:50 |
| JHTUI | yea, the big issue with octavia is no where in documentation does it state having to manually set up bridges in controllers to match the lb-management network and unless you have a good understanding of both openstack, networking, and packet sniffing to detect why you need this, people would get frustrated. I mean I'm no slouch in this and even I | 16:50 |
| JHTUI | got lost | 16:50 |
| noxoid | yea, https://shreddedbacon.com/post/openstack-kolla/ helped me piece that together | 16:50 |
| noxoid | "turned the lightbulb on" so to speak | 16:51 |
| JHTUI | yea, his last section about the provider network was the bulb that got me as far as I did. I was close, but missing some subnets | 16:51 |
| *** bengates has quit IRC | 16:57 | |
| JHTUI | what I would suggest for the ansible part of the kolla octavia buildout is a fact sheet that handles base network buildouts. For those customers/clients/other that have obtuse networks that just go beyond what you can account for, have part of the fact data sheet a bypass variable, then provide documentation on how to manually set up networking | 17:05 |
| JHTUI | for kolla octavia. Then in the kolla ansible code, you just have a block/rescue section with a when variable comparison that if 'octavia-network-manual is true' you don't execute that network block section that would set up the controllers with the bonded interfaces | 17:05 |
| *** k_mouza has quit IRC | 17:21 | |
| *** bengates has joined #openstack-kolla | 17:25 | |
| *** bengates has quit IRC | 17:40 | |
| *** rlandy has joined #openstack-kolla | 17:41 | |
| rlandy | mgoddard: hi - in kolla/docker, does the directory structure have any impact on the order in which containers are built? | 17:44 |
| *** e0ne has quit IRC | 17:58 | |
| *** dciabrin_ has joined #openstack-kolla | 18:02 | |
| *** dciabrin has quit IRC | 18:05 | |
| *** bengates has joined #openstack-kolla | 18:13 | |
| *** vishalmanchanda has quit IRC | 18:18 | |
| *** k_mouza has joined #openstack-kolla | 18:19 | |
| *** k_mouza has quit IRC | 18:20 | |
| mgoddard | rlandy: hi. No, it's determined based on the dependencies between images | 18:23 |
| mgoddard | so base first, then openstack-base, etc. | 18:24 |
| rlandy | mgoddard: ok - so a flat list of directories with the correct dependencies would work | 18:25 |
| mgoddard | rlandy: yes | 18:25 |
| rlandy | thank you | 18:25 |
| mgoddard | np | 18:25 |
| *** rlandy is now known as rlandy|brb | 18:29 | |
| *** bengates has quit IRC | 18:29 | |
| yoctozepto | hrw, mnasiadka: re: glance - I don't think glance allows to download by glance-api; I guess that is what hrw tried | 18:30 |
| yoctozepto | that said, the standalone glance client has more features; it's a bit of a concern in the community... | 18:30 |
| mnasiadka | yoctozepto: I know, it should be easy to implement in the client though | 18:37 |
| mnasiadka | But with the OSC vs glance fuss, I don’t want to even talk about it | 18:37 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: Always use upper-constraints https://review.opendev.org/715164 | 18:38 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: Always use upper-constraints https://review.opendev.org/715164 | 18:42 |
| hrw | now it will work properly | 18:42 |
| hrw | downgrade k-a and requirements, install, deploy, test, switch BOTH to master, upgrade | 18:42 |
| hrw | the side effect? we know that k-a/master installs fine with u-c/train ;D | 18:43 |
| *** pbing19 has joined #openstack-kolla | 18:46 | |
| yoctozepto | hrw: better than previously: we knew stable/train can install master... ;-) | 18:47 |
| *** rlandy|brb is now known as rlandy | 18:54 | |
| *** jbadiapa has quit IRC | 19:14 | |
| *** bengates has joined #openstack-kolla | 19:24 | |
| *** JHTUI has quit IRC | 19:27 | |
| *** cah_link has quit IRC | 19:32 | |
| *** bengates has quit IRC | 19:38 | |
| *** dougsz has quit IRC | 19:44 | |
| *** gfidente is now known as gfidente|afk | 19:49 | |
| *** jimcrowleyibm has quit IRC | 20:16 | |
| *** k_mouza has joined #openstack-kolla | 20:20 | |
| *** k_mouza has quit IRC | 20:25 | |
| *** pbing19 has quit IRC | 21:17 | |
| *** tonythomas has quit IRC | 21:41 | |
| *** todd-inmotion has joined #openstack-kolla | 22:23 | |
| *** k_mouza has joined #openstack-kolla | 22:27 | |
| *** k_mouza has quit IRC | 22:27 | |
| *** k_mouza has joined #openstack-kolla | 22:28 | |
| *** k_mouza has quit IRC | 22:33 | |
| *** TrevorV has quit IRC | 22:57 | |
| *** todd-inmotion has quit IRC | 22:57 | |
| *** gfidente|afk has quit IRC | 23:14 | |
| *** sorin-mihai has joined #openstack-kolla | 23:39 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!