| openstackgerrit | Hongbin Lu proposed openstack/kolla-ansible master: [WIP] Zun: Add zun-cni-daemon to compute node https://review.opendev.org/708213 | 00:14 |
|---|---|---|
| *** k_mouza has joined #openstack-kolla | 01:38 | |
| *** k_mouza has quit IRC | 01:43 | |
| openstackgerrit | jacky06 proposed openstack/kolla-cli master: Update hacking for Python3 https://review.opendev.org/717645 | 01:52 |
| *** sorin-mihai has quit IRC | 01:56 | |
| openstackgerrit | jacky06 proposed openstack/kolla-cli master: Fix some pep8 failed https://review.opendev.org/717650 | 02:33 |
| openstackgerrit | jacky06 proposed openstack/kolla-cli master: Fix some pep8 failed https://review.opendev.org/717650 | 02:35 |
| openstackgerrit | jacky06 proposed openstack/kolla-cli master: Use unittest.mock instead of third party mock https://review.opendev.org/717653 | 03:11 |
| openstackgerrit | jacky06 proposed openstack/kolla-cli master: Use unittest.mock instead of third party mock https://review.opendev.org/717653 | 03:13 |
| openstackgerrit | Merged openstack/kolla-cli master: Fix some pep8 failed https://review.opendev.org/717650 | 03:23 |
| *** k_mouza has joined #openstack-kolla | 03:39 | |
| *** k_mouza has quit IRC | 03:44 | |
| *** cah_link has joined #openstack-kolla | 03:44 | |
| *** cah_link1 has joined #openstack-kolla | 03:55 | |
| *** cah_link has quit IRC | 03:56 | |
| *** cah_link1 is now known as cah_link | 03:56 | |
| *** skramaja has joined #openstack-kolla | 04:07 | |
| *** ykarel|away is now known as ykarel | 04:12 | |
| openstackgerrit | jacky06 proposed openstack/kolla-cli master: Use unittest.mock instead of third party mock https://review.opendev.org/717653 | 04:13 |
| openstackgerrit | Hongbin Lu proposed openstack/kolla-ansible master: [WIP] Zun: Add zun-cni-daemon to compute node https://review.opendev.org/708213 | 04:22 |
| openstackgerrit | jacky06 proposed openstack/kolla-cli master: Use unittest.mock instead of third party mock https://review.opendev.org/717653 | 04:33 |
| *** evrardjp has quit IRC | 04:36 | |
| *** evrardjp has joined #openstack-kolla | 04:37 | |
| openstackgerrit | Hongbin Lu proposed openstack/kolla-ansible master: [WIP] Zun: Add zun-cni-daemon to compute node https://review.opendev.org/708213 | 04:40 |
| openstackgerrit | jacky06 proposed openstack/kayobe master: Use unittest.mock instead of third party mock https://review.opendev.org/717659 | 05:08 |
| *** diga has joined #openstack-kolla | 05:09 | |
| *** shyamb has joined #openstack-kolla | 05:12 | |
| openstackgerrit | jacky06 proposed openstack/kolla-cli master: Update hacking for Python3 https://review.opendev.org/717645 | 05:22 |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: Introduce /etc/timezone to Debian/Ubuntu containers https://review.opendev.org/704258 | 05:30 |
| cosmicsound | Goood day | 05:30 |
| osmanlicilegi | morning | 05:36 |
| *** shyamb has quit IRC | 05:43 | |
| openstackgerrit | Merged openstack/kolla-cli master: Use unittest.mock instead of third party mock https://review.opendev.org/717653 | 05:48 |
| *** cah_link1 has joined #openstack-kolla | 05:52 | |
| *** cah_link has quit IRC | 05:54 | |
| *** cah_link1 is now known as cah_link | 05:54 | |
| *** shyamb has joined #openstack-kolla | 05:55 | |
| openstackgerrit | jacky06 proposed openstack/kolla-cli master: Update hacking for Python3 https://review.opendev.org/717645 | 05:56 |
| openstackgerrit | jacky06 proposed openstack/kolla-cli master: Update hacking for Python3 https://review.opendev.org/717645 | 05:57 |
| *** abdysn has joined #openstack-kolla | 06:10 | |
| *** bjolo_ has joined #openstack-kolla | 06:14 | |
| bjolo_ | morning | 06:14 |
| *** bjolo_ has left #openstack-kolla | 06:16 | |
| *** bjolo_ has joined #openstack-kolla | 06:22 | |
| openstackgerrit | Merged openstack/kolla-cli master: Update hacking for Python3 https://review.opendev.org/717645 | 06:25 |
| *** jbadiapa has joined #openstack-kolla | 06:27 | |
| *** rpittau|afk is now known as rpittau | 06:35 | |
| rpittau | hrw: that was not me :) | 06:37 |
| openstackgerrit | Merged openstack/kolla-ansible master: Update hacking for Python3 https://review.opendev.org/717382 | 06:43 |
| *** iniazi has quit IRC | 06:48 | |
| *** shyam89 has joined #openstack-kolla | 06:55 | |
| *** shyamb has quit IRC | 06:58 | |
| *** zhanglong has joined #openstack-kolla | 07:03 | |
| *** bengates has joined #openstack-kolla | 07:07 | |
| *** bengates has quit IRC | 07:09 | |
| *** bengates has joined #openstack-kolla | 07:09 | |
| *** shyam89 has quit IRC | 07:11 | |
| *** e0ne has joined #openstack-kolla | 07:13 | |
| *** pbing19 has joined #openstack-kolla | 07:28 | |
| *** born2bake has joined #openstack-kolla | 07:31 | |
| *** cah_link1 has joined #openstack-kolla | 07:31 | |
| *** cah_link has quit IRC | 07:32 | |
| *** cah_link1 is now known as cah_link | 07:32 | |
| *** iniazi has joined #openstack-kolla | 07:32 | |
| *** shyamb has joined #openstack-kolla | 07:43 | |
| *** happyhemant has joined #openstack-kolla | 07:46 | |
| cosmicsound | hey osmanlicilegi | 07:48 |
| *** spsurya_ has joined #openstack-kolla | 07:59 | |
| *** ktibi has joined #openstack-kolla | 08:01 | |
| osmanlicilegi | hey cosmicsound | 08:04 |
| *** Abdallahyas has joined #openstack-kolla | 08:09 | |
| *** abdysn has quit IRC | 08:12 | |
| *** shyamb has quit IRC | 08:16 | |
| *** shyamb has joined #openstack-kolla | 08:17 | |
| hrw | rpittau: sorry | 08:20 |
| mgoddard | morning | 08:25 |
| *** priteau has joined #openstack-kolla | 08:32 | |
| priteau | Good morning Kolla folks. Please review a couple of patches which will help unbreak kayobe CI: | 08:34 |
| priteau | https://review.opendev.org/#/c/717276/ | 08:34 |
| patchbot | patch 717276 - kolla-ansible (stable/queens) - Use alternative mirrors for legacy Docker packages - 6 patch sets | 08:34 |
| priteau | https://review.opendev.org/#/c/717302/ | 08:34 |
| patchbot | patch 717302 - kolla-ansible (stable/queens) - CI: Use upper constraints to install clients - 7 patch sets | 08:34 |
| hrw | red red red all the way... | 08:37 |
| priteau | There are two separate issues, the second patch is also needed to make kolla-ansible-centos-source green | 08:39 |
| priteau | ubuntu images have other issues I haven't looked into. We don't need it for Kayobe. | 08:40 |
| hrw | ImportError: No module named cryptography.hazmat.backends | 08:41 |
| hrw | ouch | 08:42 |
| priteau | Since ubuntu jobs are non-voting as well, they could be fixed in another patch | 08:44 |
| *** zhanglong has quit IRC | 08:46 | |
| hrw | priteau: https://review.opendev.org/#/c/688745/1/tests/pre.yml may fix that crypto part | 08:49 |
| patchbot | patch 688745 - kolla-ansible (stable/rocky) - Fix CI failures (MERGED) - 1 patch set | 08:49 |
| hrw | priteau: so bindep will be run and libffi-dev installed | 08:49 |
| *** ykarel is now known as ykarel|lunch | 08:50 | |
| hrw | ignore me. looks like wheel got used. I forgot that its on x86 | 08:51 |
| *** abdysn has joined #openstack-kolla | 08:57 | |
| *** Abdallahyas has quit IRC | 08:59 | |
| *** k_mouza has joined #openstack-kolla | 09:04 | |
| *** tonythomas has joined #openstack-kolla | 09:06 | |
| cosmicsound | anyone knows how do i set the tag to latest version of horizon dashboard? | 09:07 |
| hrw | ok. rocky/stein/train/master images got published today. hope that today's recheck series will finally pass | 09:11 |
| hrw | cause it gets depressing. one CI failure takes a week to fix | 09:12 |
| *** e0ne has quit IRC | 09:13 | |
| *** e0ne has joined #openstack-kolla | 09:14 | |
| *** shyamb has quit IRC | 09:21 | |
| *** shyamb has joined #openstack-kolla | 09:22 | |
| *** vishalmanchanda has joined #openstack-kolla | 09:24 | |
| brtknr | hello ka folks, what is the variable for modifying admin and public endpoint for services? | 09:31 |
| openstackgerrit | Piotr Rabiega proposed openstack/kolla stable/train: add dpdk_telemetry and logparser, use centos-release-opstools for C8 https://review.opendev.org/717697 | 09:33 |
| priteau | brtknr: Can you be more precise? How do you want to modify them? | 09:34 |
| brtknr | So I want to use a particular interface for public and admin endpoints | 09:35 |
| brtknr | at the moment, all public, admin and internal are using the same endpoints | 09:35 |
| priteau | You're looking for this? https://docs.openstack.org/kolla-ansible/latest/admin/production-architecture-guide.html#interface-configuration | 09:36 |
| mgoddard | brtknr: why do you want to use separate networks? | 09:36 |
| *** ykarel|lunch is now known as ykarel | 09:36 | |
| *** pbing19 has quit IRC | 09:36 | |
| brtknr | mgoddard: I want keystone service to be exposed publicly so that it can be used by a standalone monasca deployment | 09:37 |
| mgoddard | brtknr: ok, sounds complicated | 09:38 |
| brtknr | mgoddard: my current guess is kolla_external_vip_interface is my friend | 09:38 |
| mgoddard | brtknr: in CI we setup a VXLAN overlay network between multinode hosts | 09:38 |
| mgoddard | that is one way to connect the bridges together | 09:38 |
| mgoddard | alternatively you could use NAT, but that might end up a bit weird | 09:39 |
| hrw | brtknr: one auth center for two separate deployments? | 09:39 |
| brtknr | hrw: yeah thats right | 09:40 |
| cosmicsound | brtknr , maybe only setting up services manually ? openstack endpoint set \ | 09:40 |
| mgoddard | brtknr: https://opendev.org/openstack/kolla-ansible/src/branch/master/tests/pre.yml#L9 | 09:40 |
| brtknr | mgoddard: that sounds more complicated than what I'm trying to do lol | 09:40 |
| mgoddard | brtknr: what you are trying to do is complicated | 09:41 |
| mgoddard | brtknr: I suppose another option is to set the node's public IP as your public interface? | 09:42 |
| mgoddard | is that what you were asking originally? | 09:42 |
| brtknr | mgoddard: yes! | 09:42 |
| mgoddard | ok | 09:42 |
| brtknr | is that a more complicated option? | 09:42 |
| mgoddard | it gets tricky with HAProxy as your cloud provider will probably filter out the VIP | 09:42 |
| cosmicsound | mgoddard , i got issue when not using haproxy my external vip wont work to link to my local. | 09:43 |
| mgoddard | so you'd need some kind of NAT to access the VIP from another host | 09:43 |
| cosmicsound | so i have to keep haproxy running to use both external and internal | 09:43 |
| mgoddard | cosmicsound: you can't use a VIP without HAProxy | 09:44 |
| cosmicsound | right | 09:44 |
| cosmicsound | and how can we expose openstack publicly without external vip | 09:44 |
| brtknr | mgoddard: am I on the right track with kolla_external_vip_interface? | 09:45 |
| cosmicsound | brtknr , if you want to allow external ip to services yes | 09:45 |
| mgoddard | cosmicsound: you have to just use a single internal IP for everything without haproxy | 09:45 |
| cosmicsound | so no external | 09:46 |
| mgoddard | brtknr: in kayobe it's public_net_name / public_interface | 09:46 |
| mgoddard | brtknr: but you will hit the problems I've described | 09:46 |
| mgoddard | brtknr: seriously I think we can setup a tunnel quickly | 09:46 |
| mgoddard | sudo ip link add vxlan0 type vxlan id 10000 local <IP of bond0> dstport 4790 | 09:48 |
| mgoddard | sudo ip link set vxlan0 mtu <bond0 MTU - 50> | 09:49 |
| mgoddard | sudo bridge fdb append 00:00:00:00:00:00 dev vxlan0 dst <IP of other node's bond0> | 09:50 |
| mgoddard | sudo ip link set vxlan0 up | 09:50 |
| mgoddard | sudo ip link set vxlan0 master breth1 | 09:50 |
| mgoddard | brtknr: ^ done | 09:50 |
| mgoddard | brtknr: make sure each host has a different IP on breth1 | 09:51 |
| mgoddard | then try pinging between the breth1 IPs | 09:51 |
| mgoddard | (you have to do that on each node) | 09:51 |
| born2bake | Hi, I am trying to add Swift(ceph rgw) to my openstack cluster. https://docs.ceph.com/docs/luminous/radosgw/keystone/ - I ve added rgw-keystone to my ceph.conf files, restarted rgw, created service, created endpoints, I can see object store tab appeared in horizon but it cant be used, cant retrieve data. I ve opened API access tab, "Object Store - " it cant see swift url I ve added. I can see swift in services, in endpoint list its | 09:52 |
| born2bake | enabled as well. | 09:52 |
| cosmicsound | born2bake , you run reconfigure? | 09:54 |
| cosmicsound | i had to run reconfigure to have it seen in horizon | 09:54 |
| cosmicsound | then its visible | 09:54 |
| born2bake | no, I didnt. | 09:54 |
| born2bake | it appeared itself after some time | 09:54 |
| cosmicsound | give it a try | 09:54 |
| cosmicsound | ok | 09:54 |
| watersj | Hello, Does masakari have nova-compute (hostmonitoring) in it? I have instance HA, where I can kill -9 qemu instance and it comes back, but if the host is killed, nothing | 09:57 |
| hrw | curl: (22) The requested URL returned error: 503 Service Unavailable | 10:01 |
| hrw | egy3e8o7t435to784to;3oT^@{"$%2%!#%$@#^%@%@ | 10:01 |
| hrw | I think that it is time for me to read more about smartnic stuff and move | 10:01 |
| hrw | openstack starts to be unfixable mess | 10:02 |
| hrw | 4 branches, 3 distros, 2 install types, 2 projects using gazillion of other projects. | 10:02 |
| hrw | kolla's recipe for doom | 10:02 |
| hrw | when something breaks in stein then stein and train are doomed. we have to fix stein, wait 1-2 days, fix train, wait 1-2 days. whole development is stuck at gerrit and everyone goes to fight with fire. | 10:04 |
| hrw | when train is doomed then master may or may not be | 10:05 |
| *** shyamb has quit IRC | 10:06 | |
| mgoddard | hrw: we're an integration project, pulling hundreds of things together. Things break | 10:10 |
| mgoddard | hrw: what do propose we should do to improve it? | 10:10 |
| *** cah_link has quit IRC | 10:11 | |
| hrw | mgoddard: I am out of ideas for any improvements | 10:13 |
| *** cah_link has joined #openstack-kolla | 10:13 | |
| hrw | amount of chained fuckups gets me closer to burnout then usual | 10:14 |
| mgoddard | hrw: we have ideas about primary/secondary/tertiary images, so that we can ignore failure of tertiary images | 10:15 |
| mgoddard | hrw: we just need someone to implement it | 10:15 |
| mgoddard | hrw: master branch has been pretty stable, and that is where our development is done | 10:16 |
| hrw | mgoddard: to get some changes into master I/we need working train. which needs working stein. which needs working rocky. | 10:17 |
| *** shyamb has joined #openstack-kolla | 10:17 | |
| mgoddard | hrw: not true. branches only depend on the previous release | 10:17 |
| mgoddard | hrw: master is working fine | 10:17 |
| mgoddard | train is only blocked on upgrade due to stein | 10:18 |
| hrw | mgoddard: https://review.opendev.org/713854 needs train images and https://review.opendev.org/716574 to land in train. | 10:18 |
| patchbot | patch 713854 - kolla-ansible - CI: add Debian source upgrade job on AArch64 - 3 patch sets | 10:18 |
| patchbot | patch 716574 - kolla (stable/train) - CI: Use network mode = host - 2 patch sets | 10:18 |
| hrw | kolla-ansible-ubuntu-source-upgrade in train is broken | 10:19 |
| mgoddard | ok, but that's only blocked because you're patching train | 10:19 |
| mgoddard | anyway | 10:20 |
| mgoddard | I hear your point. We spend a lot of time fighting CI fires | 10:20 |
| mgoddard | and yes it's a PITA | 10:20 |
| hrw | I am out of ideas for any development on master | 10:20 |
| mgoddard | hrw: we have ideas. If you have time to implement them, that would be great | 10:21 |
| hrw | and with release being so close I wonder how many months/weeks we will be delayed with ussuri release | 10:21 |
| hrw | mgoddard: any new ideas should be now postponed to Vicky | 10:21 |
| hrw | or whatever V was named | 10:22 |
| mgoddard | yes, but they can still be worked on | 10:22 |
| hrw | we need to get master shiny and ready. without being loaded with work on older releases | 10:22 |
| hrw | and I have kevinz and xinliang working on rocky->stein->train upgrade so I want train->ussuri upgrade job working on aarch64 so next time they will get something rather working than maybe working | 10:23 |
| brtknr | mgoddard: so same on the other host? | 10:24 |
| mgoddard | brtknr: yes, but swap IPs around | 10:24 |
| hrw | anyway, I have some online training to finish so end of kolla's work for me today | 10:24 |
| hrw | INFO:kolla.common.utils.cinder-volume:1mhttp://mirror.iad.rax.openstack.org/centos/7/cloud/x86_64/openstack-train/Packages/p/python2-cinderclient-5.0.1-1.el7.noarch.rpm: [Errno 14] HTTP Error 404 - Not Found | 10:25 |
| hrw | heh. infra out of sync | 10:25 |
| mgoddard | hrw: let me know if you want to talk about potential improvements | 10:28 |
| hrw | mgoddard: I got proposals to move to work on IoT/Edge or SmartNICs. | 10:29 |
| brtknr | mgoddard: ok they seem to be talking to each other | 10:29 |
| brtknr | but how do i verify them via tcpdump? | 10:29 |
| mgoddard | brtknr: great :) | 10:29 |
| brtknr | tcpdump -n icmp? | 10:30 |
| mgoddard | brtknr: why tcpdump? | 10:30 |
| brtknr | only thing i could think of | 10:30 |
| brtknr | as in, if i set the interface down, i cant ping the other node | 10:30 |
| yoctozepto | hey fellow kollars, I will most likely (as in: I hope) be able to wear my firefighting equipment later, most probably this evening | 10:31 |
| *** sean-k-mooney has joined #openstack-kolla | 10:34 | |
| *** hjensas|afk is now known as hjensas | 10:55 | |
| watersj | what generates COMPUTE_HOST failure notification? | 11:01 |
| *** shyamb has quit IRC | 11:02 | |
| hrw | ok, so who has any idea why horizon failes in train->stein upgrade jobs? | 11:11 |
| mgoddard | hrw: not sure. Just pulling down a stein horizon image | 11:13 |
| mgoddard | hrw: docker logs look fine. No horizon logs though. HAProxy says 503 | 11:13 |
| watersj | If I manually create COMPUTE_HOST notification, masakari goes ahead and evacuates the instance and powers it on. Is haproxy supposed to create notification when host goes down? | 11:15 |
| hrw | mgoddard: that's the problem | 11:15 |
| hrw | anyway... k-a in stein is borned | 11:15 |
| hrw | https://review.opendev.org/#/c/717259 got another recheck | 11:16 |
| patchbot | patch 717259 - kolla-ansible (stable/stein) - CI: Use upper constraints to install clients - 3 patch sets | 11:16 |
| hrw | until it gets in stein is dead | 11:16 |
| hrw | mgoddard: when do you plan to make new stein/train releases? | 11:17 |
| *** shyamb has joined #openstack-kolla | 11:17 | |
| mgoddard | hrw: no plan currently but can do if someone needs them | 11:17 |
| mgoddard | hrw: also, doesn't have to be me | 11:18 |
| hrw | mgoddard: I think that once we get them green on ci then would be good to release | 11:18 |
| mgoddard | sure | 11:18 |
| hrw | as they will have 'use u-c' patch in every branch | 11:18 |
| hrw | it would cut potencial support question | 11:19 |
| hrw | s | 11:19 |
| mgoddard | +1 | 11:19 |
| *** zhanglong has joined #openstack-kolla | 11:25 | |
| mgoddard | hrw: managed to run stein horizon locally. Let's see what zuul says | 11:28 |
| mgoddard | although train jobs have already failed | 11:29 |
| *** JamesBenson has joined #openstack-kolla | 11:36 | |
| mgoddard | looks like our disk is filling up in the centos8 ceph-ansible upgrade job: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dc0/694799/36/check/kolla-ansible-centos8-source-upgrade-ceph-ansible/dc059bb/primary/logs/system_logs/df.txt | 11:39 |
| *** shyamb has quit IRC | 11:44 | |
| hrw | https://review.opendev.org/#/c/717259/ - k-a-ubuntu-{binary,source} look like not even started horizon... | 11:51 |
| patchbot | patch 717259 - kolla-ansible (stable/stein) - CI: Use upper constraints to install clients - 3 patch sets | 11:51 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/rocky: Make deploy jobs voting again https://review.opendev.org/717726 | 11:52 |
| hrw | +2 | 11:53 |
| mgoddard | hrw: https://9b719019225fc3f1a633-9c3ff8aa7dcabcee88b15fa3c77a7faa.ssl.cf2.rackcdn.com/717259/3/check/kolla-ansible-ubuntu-source/afd3ae4/primary/logs/docker_logs/horizon.txt | 11:53 |
| mgoddard | hrw: although, now I look at that log I can see it is still compressing static files | 11:53 |
| mgoddard | hrw: and when I started the container locally it took AGES to compress | 11:54 |
| mgoddard | hrw: so maybe we need to wait longer? | 11:54 |
| *** shyamb has joined #openstack-kolla | 11:56 | |
| mgoddard | hrw: took over a minute on my laptop, probably longer in CI | 11:57 |
| mgoddard | hrw: :o using 100% CPU | 11:57 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: CI: Use upper constraints to install clients https://review.opendev.org/717259 | 11:57 |
| mgoddard | ^ bumped horizon timeout to 300s, let's see how it goes | 11:58 |
| hrw | mgoddard: why we not compress that during build... | 12:00 |
| *** gfidente has joined #openstack-kolla | 12:07 | |
| *** shyam89 has joined #openstack-kolla | 12:11 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla master: Upgrade RabbitMQ to 3.8 https://review.opendev.org/694799 | 12:14 |
| *** shyamb has quit IRC | 12:15 | |
| *** priteau has quit IRC | 12:35 | |
| *** shyam89 has quit IRC | 12:43 | |
| *** Luzi has joined #openstack-kolla | 13:01 | |
| mgoddard | hrw: good question | 13:04 |
| mgoddard | hrw: after 7 minutes container uptime, horizon still compressing: | 13:04 |
| mgoddard | root 31399 31333 31399 99.1 3.7 336828 302972 /var/lib/kolla/venv/bin/python /var/lib/kolla/venv/bin/manage.py compress --force | 13:04 |
| mgoddard | hrw: maybe due to runtime customisation we need to compress at runtime? | 13:05 |
| *** ykarel is now known as ykarel|afk | 13:11 | |
| *** jimcrowleyibm has joined #openstack-kolla | 13:33 | |
| *** also_stingrayza is now known as stingrayza | 13:39 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: CI: Use upper constraints to install clients https://review.opendev.org/717259 | 13:39 |
| hrw | mgoddard: maybe add 10s delay between attempts? | 13:41 |
| mgoddard | hrw: already do | 13:42 |
| hrw | ops, right ;d | 13:42 |
| mgoddard | generalfuzz: around? | 13:43 |
| *** Blinkiz has joined #openstack-kolla | 13:53 | |
| *** zhanglong has quit IRC | 13:54 | |
| *** TrevorV has joined #openstack-kolla | 13:56 | |
| *** Blinkiz has quit IRC | 13:57 | |
| *** cdearborn has joined #openstack-kolla | 14:00 | |
| *** Blinkiz has joined #openstack-kolla | 14:01 | |
| *** lennyb has quit IRC | 14:04 | |
| *** ykarel|afk is now known as ykarel | 14:06 | |
| *** Luzi has quit IRC | 14:15 | |
| *** gary_perkins has quit IRC | 14:30 | |
| *** gary_perkins has joined #openstack-kolla | 14:32 | |
| born2bake | Hi, I am trying to add Swift(ceph rgw) to my openstack cluster. I ve added rgw-keystone to my ceph.conf files, restarted rgw, created service, created endpoints, I can see object store tab appeared in horizon but it cant be used - https://pastebin.com/qKJ83QM9 AccessDenied error | 14:33 |
| born2bake | any ideas on that one? | 14:33 |
| watersj | not familiar with swift but to get external ceph working I needed to export my ceph.client.cinder.keyring and like keys into their corresponding /etc/kolla/config/[cinder,nova,...] directories | 14:42 |
| *** bengates_ has joined #openstack-kolla | 14:46 | |
| *** bengates has quit IRC | 14:47 | |
| kplant | born2bake: what did you use for the endpoints | 14:54 |
| kplant | i have ceph-rgw backed swift working in mine | 14:54 |
| born2bake | openstack endpoint create --region RegionOne swift public http://10.0.2.1:8080/swift/v1 | 14:55 |
| born2bake | openstack service create --name=swift \ | 14:55 |
| born2bake | --description="Swift Service" \ | 14:55 |
| born2bake | object-store | 14:55 |
| born2bake | curl http://10.0.2.1:8080 | 14:55 |
| born2bake | <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult> | 14:55 |
| kplant | ah | 14:56 |
| born2bake | curl http://10.0.2.1:8080/swift/v1 - NoSuchKey | 14:56 |
| kplant | you need /swift/v1/AUTH_$(tenant_id)s | 14:56 |
| born2bake | need that where? in endpoints? | 14:56 |
| kplant | yes | 14:56 |
| born2bake | delete my previous one first right? | 14:56 |
| kplant | yes | 14:56 |
| born2bake | thank you mate | 14:56 |
| kplant | yw | 14:57 |
| kplant | let me know how it goes | 14:57 |
| noxoid | ive always been curious why AUTH_ is needed. it works fine for me without it | 14:58 |
| kplant | depends on your ceph config | 14:58 |
| kplant | there's a line to use the keystone tenant id as the ceph "user" | 14:59 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/stein: DNM: Test horizon & pyScss==1.3.4 compress https://review.opendev.org/717776 | 15:00 |
| noxoid | rgw_keystone_implicit_tenants ? | 15:00 |
| noxoid | or rgw_swift_account_in_url | 15:00 |
| born2bake | I dont have it "keystone tenant id" in my ceph.conf | 15:01 |
| born2bake | running reconfigure, will see how it goes | 15:01 |
| mgoddard | hrw: horizon works with a timeout of 1000s! | 15:03 |
| mgoddard | looks like pyScss==1.3.7 might be the culprit | 15:03 |
| cosmicsound | what pools you folks have for rgw in ceph for swift | 15:04 |
| hrw | mgoddard: o! | 15:11 |
| hrw | mgoddard: for py2 1.3.4 is fine | 15:11 |
| hrw | mgoddard: py3 will fetch setuptools 46+ so 1.3.7 is needed | 15:11 |
| mgoddard | hrw: hmm, maybe 1.3.7 doesn't work well with py2? | 15:12 |
| mgoddard | because we don't seem to hit this on master | 15:12 |
| hrw | mgoddard: master uses 1.3.7 | 15:13 |
| hrw | mgoddard: horizon in master uses 1.3.7, in train iirc too | 15:14 |
| mgoddard | hrw: right, but also py3 | 15:14 |
| hrw | mgoddard: pyscss 1.3.6 was '1.3.5 + setuptools v46+', 1.3.7 was 'let build without compiler' | 15:14 |
| mgoddard | hrw: centos7 train image has 1.3.4 | 15:14 |
| hrw | mgoddard: so 1.3.7 can be used with py2 but horizon in stein- stick to 1.3.4 | 15:15 |
| mgoddard | hrw: 1.3.7 is already in stein upper-constraints | 15:15 |
| *** tonythomas has quit IRC | 15:15 | |
| hrw | o. | 15:16 |
| *** abdysn has quit IRC | 15:16 | |
| hrw | still fine with py2/py3 | 15:16 |
| hrw | mgoddard: ok, I know then what we may need to fix to get 1.3.7 pyscss faster. have to check logs | 15:17 |
| hrw | INFO:kolla.common.utils.horizon: Created wheel for pyScss: filename=pyScss-1.3.7-cp27-cp27mu-linux_x86_64.whl | 15:19 |
| hrw | so it built that faster part | 15:19 |
| born2bake | kplant didnt help :/ | 15:22 |
| kplant | can i see your ceph rgw config? | 15:24 |
| kplant | the "rgw keystone *" lines | 15:26 |
| *** ykarel is now known as ykarel|away | 15:28 | |
| noxoid | cosmicsound: if you give the radosgw daemon a key with the write mon permission (iirc) itll create the pools necessary itself | 15:29 |
| noxoid | unless you're talking about if we use erasure/replication | 15:30 |
| openstackgerrit | Merged openstack/kolla stable/rocky: Make deploy jobs voting again https://review.opendev.org/717726 | 15:31 |
| *** vishalmanchanda has quit IRC | 15:33 | |
| cosmicsound | noxoid , i was thinnking i miss some pools, i was able to create a bucker, yet it failed when tried to create a folder in it | 15:33 |
| cosmicsound | https://mdb.uhlhost.net/uploads/078780d235c83ae4/image.png | 15:34 |
| noxoid | may want to try creating it from the cli and see what error its actually returning | 15:35 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: CI: Bump dashboard access timeout to 1000s https://review.opendev.org/717782 | 15:37 |
| cosmicsound | noxoid , will try that | 15:46 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Add kolla_enable_openstack_core variable https://review.opendev.org/717785 | 15:49 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/stein: DNM: Test horizon & pyScss==1.3.4 compress https://review.opendev.org/717776 | 15:50 |
| mgoddard | hrw, yoctozepto, osmanlicilegi: could use a review of https://review.opendev.org/#/c/717276 which blocks kayobe CI | 15:51 |
| patchbot | patch 717276 - kolla-ansible (stable/queens) - Use alternative mirrors for legacy Docker packages - 6 patch sets | 15:51 |
| mgoddard | hrw, yoctozepto, osmanlicilegi: also https://review.opendev.org/#/c/717259 will unblock stein | 15:53 |
| patchbot | patch 717259 - kolla-ansible (stable/stein) - CI: Use upper constraints to install clients - 5 patch sets | 15:53 |
| mgoddard | and https://review.opendev.org/717782 should unblock train | 15:53 |
| patchbot | patch 717782 - kolla-ansible (stable/train) - CI: Bump dashboard access timeout to 1000s - 1 patch set | 15:53 |
| *** tonythomas has joined #openstack-kolla | 16:00 | |
| osmanlicilegi | mgoddard: done | 16:02 |
| mgoddard | thanks osmanlicilegi | 16:03 |
| *** rpittau is now known as rpittau|afk | 16:08 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/queens: Use alternative mirrors for legacy Docker packages https://review.opendev.org/717276 | 16:14 |
| openstackgerrit | Merged openstack/kolla-ansible stable/queens: CI: Use upper constraints to install clients https://review.opendev.org/717302 | 16:14 |
| openstackgerrit | Piotr Rabiega proposed openstack/kolla stable/train: add dpdk_telemetry and logparser for C8, use centos-release-opstools for C8 https://review.opendev.org/717697 | 16:27 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Add docs and release note for CentOS 8 https://review.opendev.org/717796 | 16:30 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Add kolla_enable_openstack_core variable https://review.opendev.org/717785 | 16:32 |
| *** evrardjp has quit IRC | 16:36 | |
| *** evrardjp has joined #openstack-kolla | 16:37 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Add docs and release note for CentOS 8 https://review.opendev.org/717796 | 16:37 |
| yoctozepto | mgoddard: so stein has newer pyscss installed? | 16:39 |
| mgoddard | yoctozepto: yep! | 16:39 |
| yoctozepto | mgoddard: woot | 16:39 |
| yoctozepto | ;D | 16:39 |
| mgoddard | well thankfully it did or I might not have found a diff | 16:40 |
| yoctozepto | whatever, deprecate horizon | 16:40 |
| kplant | lol | 16:41 |
| yoctozepto | kplant: what? it's clumsy | 16:41 |
| kplant | i agree | 16:41 |
| kplant | i don't even know what's good for web anymore | 16:44 |
| kplant | but it's definitely not django | 16:44 |
| *** k_mouza has quit IRC | 16:45 | |
| *** skramaja has quit IRC | 16:52 | |
| openstackgerrit | Pedro Henrique Pereira Martins proposed openstack/kolla-ansible master: Add suport to OpenID Connect Authentication flow https://review.opendev.org/695432 | 16:54 |
| *** k_mouza has joined #openstack-kolla | 16:57 | |
| yoctozepto | kplant: django or not, openstack begs to be consumed via browser-side requests to services APIs, not clumsy middleman | 16:58 |
| *** e0ne has quit IRC | 16:59 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: Add docs and release note for CentOS 8 https://review.opendev.org/717807 | 17:04 |
| kplant | then horizon could turn into a little turd service that just fires js to the client | 17:07 |
| kplant | is that what you're saying? | 17:07 |
| mgoddard | hrw, yoctozepto: verified, PyScss 1.3.4 in stein reverts to sane compression time: https://ac68d2db0e7c0ff52223-7f8a4c585e620728c399ab8dbffa7ef2.ssl.cf2.rackcdn.com/717776/2/check/kolla-ansible-centos-source/2272ca9/primary/logs/ansible/test-openstack | 17:07 |
| *** k_mouza has quit IRC | 17:07 | |
| mgoddard | tested via https://review.opendev.org/717776 | 17:08 |
| patchbot | patch 717776 - kolla (stable/stein) - DNM: Test horizon & pyScss==1.3.4 compress - 2 patch sets | 17:08 |
| *** bengates has joined #openstack-kolla | 17:10 | |
| *** bengates_ has quit IRC | 17:14 | |
| hrw | mgoddard: can you create an issue on https://github.com/Kronuz/pyScss tracker? | 17:17 |
| * hrw off | 17:18 | |
| mgoddard | hrw: https://github.com/Kronuz/pyScss/issues/398 | 17:19 |
| mgoddard | just linked to our issue | 17:19 |
| yoctozepto | kplant: another spin-off project but yeah | 17:19 |
| kplant | that would be great | 17:20 |
| kplant | is there any function that _needs_ to hit the admin endpoint? | 17:21 |
| *** k_mouza has joined #openstack-kolla | 17:23 | |
| *** k_mouza has quit IRC | 17:27 | |
| *** bengates has quit IRC | 17:29 | |
| *** ktibi has quit IRC | 17:34 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: CI: Use upper constraints to install clients https://review.opendev.org/717259 | 17:43 |
| born2bake | cosmicsound object store disappeared in horizon for me :) what you did to make it back? | 17:44 |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend Keystone HAProxy traffic https://review.opendev.org/713986 | 17:48 |
| cosmicsound | born2bake , well until now nothing :D | 17:49 |
| cosmicsound | since it never dissapeared yet | 17:49 |
| cosmicsound | just check make sure you have properly made the service urls | 17:50 |
| cosmicsound | public admin internal | 17:50 |
| yoctozepto | mgoddard: wdyt regarding zun cni as highlight? | 17:57 |
| *** k_mouza has joined #openstack-kolla | 17:58 | |
| *** crindi_ has joined #openstack-kolla | 18:04 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/train: Fix ovs fw driver for the other ovs agent https://review.opendev.org/716175 | 18:05 |
| *** crindi has quit IRC | 18:08 | |
| *** thatcher has quit IRC | 18:08 | |
| *** crindi_ is now known as crindi | 18:08 | |
| *** admin0 has quit IRC | 18:09 | |
| *** k_mouza has quit IRC | 18:16 | |
| *** diga has quit IRC | 18:19 | |
| *** gfidente is now known as gfidente|afk | 18:42 | |
| *** kukacz has quit IRC | 18:55 | |
| *** kukacz has joined #openstack-kolla | 18:57 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: add dpdk_telemetry and logparser for C8, use centos-release-opstools for C8 https://review.opendev.org/717697 | 18:58 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: CI: Bump dashboard access timeout to 1000s https://review.opendev.org/717782 | 18:59 |
| openstackgerrit | Merged openstack/kolla master: Upgrade RabbitMQ to 3.8 https://review.opendev.org/694799 | 19:16 |
| *** dking_desktop has quit IRC | 19:20 | |
| born2bake | kplant nah, its still failing.. tried everything | 19:24 |
| born2bake | https://pastebin.com/QrUWPCNw | 19:24 |
| born2bake | same thing | 19:24 |
| kplant | that's so weird | 19:25 |
| kplant | can i get your output from `openstack endpoint list --service swift` ? | 19:25 |
| cosmicsound | how do i create buckets via cli | 19:26 |
| cosmicsound | in swift | 19:26 |
| kplant | it's a container | 19:26 |
| kplant | openstack container create | 19:26 |
| born2bake | http://paste.openstack.org/show/791700/ | 19:27 |
| kplant | you're missing AUTH_$(tenant_id)s | 19:27 |
| kplant | you can hit the endpoint manually | 19:27 |
| kplant | get the uuid of your admin project | 19:27 |
| kplant | and plug it in yourself with curl | 19:27 |
| kplant | should work | 19:27 |
| born2bake | I had it before when I ve added those tenant things and it didnt work...so I decided to revert it | 19:28 |
| born2bake | even without tenant you can see in the log - https://pastebin.com/QrUWPCNw | 19:28 |
| born2bake | it goest the same way without swift url = true | 19:28 |
| kplant | is your rgw endpoint reachable from keystone? | 19:28 |
| kplant | more importantly, is keystone reachable from your rgw(s) | 19:29 |
| born2bake | curl swift/v1 - 404, curl swift/v1/AUTH_tenantid - HTTP/1.1 301 Moved Permanently | 19:30 |
| born2bake | <kplant> is your rgw endpoint reachable from keystone? - how do I check that? from keystone container? | 19:31 |
| kplant | you can check from the host the container is on | 19:31 |
| kplant | http://paste.openstack.org/show/pxgWwjK8f8mcrZSarC9Y/ | 19:32 |
| kplant | i have :8443 as a service in haproxy for https and loadbalancing | 19:32 |
| kplant | other than that, it's the same | 19:32 |
| born2bake | I can curl my rgw from keystone container, from host | 19:35 |
| born2bake | and I can see requests in rgw logs. | 19:36 |
| kplant | with your accepted roles only being "admin" | 19:36 |
| kplant | i assume the user you're trynig with has the admin role for that project? | 19:36 |
| born2bake | . /etc/kolla/admin-(post-deploy-script) | 19:37 |
| born2bake | so I ve added same admin and password in rgw keystone | 19:38 |
| *** keper7 has joined #openstack-kolla | 19:40 | |
| kplant | i don't know what else to say really, i just created the swift service, those three endpoints | 19:43 |
| kplant | configured the rgws | 19:43 |
| kplant | and it worked | 19:43 |
| kplant | then i made it fancy with haproxy | 19:43 |
| born2bake | what output do you have when you curl your swift? | 19:57 |
| openstackgerrit | Dincer Celik proposed openstack/kolla master: Upgrade Elasticsearch and Kibana to 6.x https://review.opendev.org/647748 | 20:07 |
| openstackgerrit | Dincer Celik proposed openstack/kolla master: Follow up on RabbitMQ 3.8 upgrade https://review.opendev.org/717845 | 20:14 |
| bjolo_ | mgoddard: i just saw you mention centos8 containters for train 9.1 release. when is that release coming? | 20:17 |
| yoctozepto | bjolo_: "soon" :-) but you can grab kolla from stable/train branch as it's quite... well... stable :-) | 20:22 |
| *** cah_link has quit IRC | 20:42 | |
| *** e0ne has joined #openstack-kolla | 20:50 | |
| openstackgerrit | Jason Anderson proposed openstack/kolla-ansible master: [elasticsearch] Update config for 6.x https://review.opendev.org/709624 | 20:59 |
| openstackgerrit | Jason Anderson proposed openstack/kolla-ansible master: [elasticsearch] Add migration for Kibana 6.x index https://review.opendev.org/709625 | 20:59 |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe stable/queens: Fix multiple CI failures https://review.opendev.org/717208 | 21:00 |
| openstackgerrit | Jason Anderson proposed openstack/kolla-ansible master: [elasticsearch] Add migration for Kibana 6.x index https://review.opendev.org/709625 | 21:09 |
| *** e0ne has quit IRC | 21:16 | |
| *** JamesBenson has quit IRC | 22:00 | |
| *** k_mouza has joined #openstack-kolla | 22:16 | |
| *** born2bake has quit IRC | 22:17 | |
| *** k_mouza has quit IRC | 22:21 | |
| *** TrevorV has quit IRC | 22:24 | |
| *** k_mouza has joined #openstack-kolla | 22:46 | |
| *** k_mouza has quit IRC | 22:47 | |
| *** k_mouza has joined #openstack-kolla | 22:48 | |
| *** k_mouza has quit IRC | 22:52 | |
| *** jimcrowleyibm has quit IRC | 23:05 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
