| *** gyee has quit IRC | 00:08 | |
| snadge | yeah i decided to plod ahead with a 3 node Centos 7 test deployment | 00:22 |
|---|---|---|
| snadge | but in prod we use RHEL 7.. figured RHEL 8 will likely have improvements with regards to containers, virtualisation etc | 00:22 |
| snadge | im excited that the motivation to use the standard openstack ansible deployment scripts in production has been kicked into gear | 00:26 |
| snadge | new shiny ;) | 00:27 |
| openstackgerrit | Merged openstack/openstack-ansible-os_panko master: Add global override for service bind address https://review.opendev.org/683145 | 00:40 |
| *** markvoelker has quit IRC | 01:35 | |
| *** spsurya has joined #openstack-ansible | 01:47 | |
| *** Smeared_Beard has quit IRC | 02:04 | |
| *** irclogbot_1 has quit IRC | 02:09 | |
| *** irclogbot_1 has joined #openstack-ansible | 02:11 | |
| *** cp has quit IRC | 02:54 | |
| *** cp has joined #openstack-ansible | 02:56 | |
| snadge | im trying to boostrap 19.0.4 and its bombing out with the error "You are using pip version 9.0.1, however version 19.2.3 is available." | 03:07 |
| snadge | i haven't seen this before.. latest centos 7 | 03:07 |
| snadge | /opt/ansible-runtime/bin/pip install --upgrade pip | 03:08 |
| snadge | that seems to pretend to upgrade it.. but then bails out with the same error, so i'm obviously missing something | 03:09 |
| *** markvoelker has joined #openstack-ansible | 03:10 | |
| *** BjoernT has quit IRC | 03:16 | |
| snadge | HTTPError: 403 Client Error: Forbidden for url: http://opendev.org/openstack/requirements/raw/238bb754fb637c9f548b03af964f4e882d806b75/upper-constraints.txt | 03:20 |
| snadge | ahh im guessing that's the actual problem.. never mind | 03:21 |
| *** Jeffrey4l has quit IRC | 03:38 | |
| *** Jeffrey4l has joined #openstack-ansible | 03:40 | |
| *** alvinstarr has quit IRC | 04:38 | |
| *** Miouge has quit IRC | 04:40 | |
| *** dasp has quit IRC | 04:40 | |
| *** jamesdenton has quit IRC | 04:41 | |
| *** Miouge has joined #openstack-ansible | 04:42 | |
| *** jamesdenton has joined #openstack-ansible | 04:44 | |
| *** Smeared_Beard has joined #openstack-ansible | 04:46 | |
| *** pcaruana has joined #openstack-ansible | 04:52 | |
| *** Smeared_Beard has quit IRC | 05:20 | |
| *** SmearedBeard has joined #openstack-ansible | 05:39 | |
| *** kopecmartin|off is now known as kopecmartin | 05:49 | |
| jrosser | morning | 06:21 |
| *** cshen has joined #openstack-ansible | 06:28 | |
| *** luksky has joined #openstack-ansible | 06:35 | |
| *** dasp has joined #openstack-ansible | 06:40 | |
| *** jbadiapa has joined #openstack-ansible | 07:00 | |
| *** jbadiapa has quit IRC | 07:13 | |
| *** tosky has joined #openstack-ansible | 07:17 | |
| *** jbadiapa has joined #openstack-ansible | 07:19 | |
| ChosSimbaOne | morning. | 07:30 |
| ChosSimbaOne | noonedeadpunk: Did you setup your cors for gnoochi + grafana via the playbooks? I do not see any variables relating to this? | 07:31 |
| *** jbadiapa has quit IRC | 07:34 | |
| *** ivve has joined #openstack-ansible | 07:56 | |
| *** rpittau|afk is now known as rpittau | 07:57 | |
| *** markvoelker has quit IRC | 08:09 | |
| noonedeadpunk | ChosSimbaOne: I just used overrides for this | 08:33 |
| noonedeadpunk | jrosser: seems that things passing, great work! | 08:37 |
| noonedeadpunk | And even https://review.opendev.org/#/c/681002/ is ok | 08:37 |
| jrosser | yeah, its looking good | 08:38 |
| jrosser | we have one little problem i'm just patching | 08:38 |
| *** jawad_axd has joined #openstack-ansible | 08:38 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_nova master: Ensure git is available to clone novnc or spice https://review.opendev.org/686104 | 08:39 |
| ChosSimbaOne | noonedeadpunk: Ahh, okay will look into this. | 08:39 |
| jrosser | ^ that fixes the big cleanup of nova packages that merged yesterday | 08:40 |
| jrosser | it's another problem that we only test roles with metal jobs | 08:40 |
| jrosser | and that breaks only for container jobs on the integrated repo test | 08:40 |
| noonedeadpunk | yeah, I feel like lxc testing is pretty important | 08:41 |
| noonedeadpunk | but it takes so much time for integrated builds | 08:41 |
| jrosser | everything on one metal host leaves a lot of room for package installs to be broken, even for multinode metal use case | 08:42 |
| jrosser | so lxc job is still a good proxy for a multinode metal deployment | 08:42 |
| noonedeadpunk | and we're testing only kvm, while we eventually might drop packages necessary for other virts.. | 08:42 |
| jrosser | then theres one more thing broken | 08:43 |
| jrosser | debian jobs failed but i wonder if thats an infra mirror thing | 08:43 |
| jrosser | https://zuul.opendev.org/t/openstack/build/5b184b93fd2442ba86709ea14c8f7b54/log/job-output.txt#2937 | 08:44 |
| noonedeadpunk | It looks like not fully synced repos | 08:45 |
| jrosser | yes because https://packages.debian.org/stretch/libssl-dev | 08:45 |
| jrosser | that looks consistent wanting 1.1.0l-1~deb9u1 version for both the lib and the -dev package | 08:45 |
| noonedeadpunk | yep, exactly | 08:46 |
| noonedeadpunk | So I expect that recheck will eventually work | 08:46 |
| jawad_axd | jrosser: noonedeadpunk: Quick question: In HA haproxy nodes(3), if internal load balancer IP is e.g 10.10.25.100, Do I need to setup this IP on all three haproxy nodes so it will work with failover while haproxy nodes has IPs 10.10.25.[90,91,92]? | 08:54 |
| jrosser | jawad_axd: no, you let keepalived deal with that IP | 08:54 |
| jawad_axd | So you mean, I dont really need separate nice for internal lb? | 08:55 |
| jawad_axd | *nic | 08:55 |
| jrosser | you can have as many/few nic as you want | 08:56 |
| jawad_axd | Do I need to define internal lb IP on some nic or OSA takes care of it automatically ? | 08:57 |
| jrosser | look at the defaults for the haproxy role | 08:59 |
| jrosser | here is a quick grep of my user variables for keepalived stuff http://paste.openstack.org/show/780656/ | 08:59 |
| * jrosser got a couple hours meetings now, bbl | 08:59 | |
| ChosSimbaOne | jrosser: enjoy :-) | 09:00 |
| cshen | noonedeadpunk I saw you also commented here https://bugs.launchpad.net/cinder/+bug/1806156 , do you have any workaround at the end? | 09:23 |
| openstack | Launchpad bug 1806156 in Cinder "shared_targets_online_data_migration fails when cinder-volume service not running" [Undecided,Confirmed] | 09:23 |
| noonedeadpunk | not really... But haven't experience problems on the fresh installs... | 09:25 |
| cshen | noonedeadpunk Ok, it's strange. We were also hit by fresh install yesterday, so not only by upgrade. | 09:26 |
| noonedeadpunk | yep, I saw your comment... That's sad. | 09:27 |
| noonedeadpunk | Btw, why don't you run stein fresh install?:) | 09:28 |
| cshen | OH, you mean fresh install of stein? | 09:28 |
| cshen | Actually we're still using Rocky in the production. So I was testing fresh install of OSA 18.1.8. | 09:29 |
| noonedeadpunk | yep | 09:29 |
| noonedeadpunk | Ah, it's test deployment | 09:29 |
| cshen | noonedeadpunk: Ok, thanks for the info. I need to find workaround for Rocky myself. | 09:30 |
| *** jbadiapa has joined #openstack-ansible | 09:30 | |
| noonedeadpunk | yeah, sorry, really hadn't enaough time for digging more into that:( | 09:34 |
| cshen | noonedeadpunk: no problem. | 09:34 |
| cshen | noonedeadpunk: fyi, I found somebody removed cinder online data migration in Train. https://opendev.org/openstack/cinder/commit/230bda82c1ada91dcec90b21913d713795164b19 Maybe it helps you as well. | 09:37 |
| noonedeadpunk | cshen: so it looks like shared_targets_online_data_migration should be removed in rocky at all | 09:40 |
| noonedeadpunk | Since they were added in Queens | 09:40 |
| cshen | noonedeadpunk: at least for the fresh install, there is NO need to do any online data migration. | 09:41 |
| admin0 | what is the best way to delete instances which has attachments ? shutdown, unattach and then delete instance and volume ? | 09:41 |
| admin0 | my issue is that the instances can be deleted, leaving cinder hanging | 09:41 |
| cshen | noonedeadpunk: for Q->R, I think it's not necessary either. | 09:41 |
| noonedeadpunk | cshen: actually they shouldn't fail even if they are runned | 09:41 |
| noonedeadpunk | admin0: so cinder thinks that volume is attached even after deleting instance? | 09:42 |
| cshen | cshen: for online data migration, the step needs RPC communication, but during the upgrade, you can't rely on every cinder service is up and running. | 09:45 |
| cshen | noonedeadpunk: so the online data migration task is fundamentally flawed. | 09:52 |
| noonedeadpunk | but cinder folks in the bug seemed to have a bit another opinion on things:) | 09:53 |
| jrosser | during upgrade don't we serialise these tasks | 09:58 |
| *** hwoarang has quit IRC | 10:01 | |
| *** hwoarang has joined #openstack-ansible | 10:02 | |
| *** markvoelker has joined #openstack-ansible | 10:10 | |
| *** markvoelker has quit IRC | 10:15 | |
| *** cshen has quit IRC | 10:26 | |
| *** SmearedBeard has quit IRC | 10:27 | |
| *** ivve has quit IRC | 10:32 | |
| *** zbr|ruck is now known as zbr|lunch | 10:39 | |
| *** cshen has joined #openstack-ansible | 10:43 | |
| *** tosky_ has joined #openstack-ansible | 10:49 | |
| *** tosky has quit IRC | 10:52 | |
| *** jbadiapa has quit IRC | 10:57 | |
| *** tosky_ is now known as tosky | 10:58 | |
| tacco | is there any way to integrate existing ceph cluster in osa without ssh-access to the MONs? | 10:59 |
| tacco | didn't found anything in the ceph_client role | 10:59 |
| *** tosky_ has joined #openstack-ansible | 11:02 | |
| *** tosky is now known as Guest18639 | 11:02 | |
| *** tosky_ is now known as tosky | 11:02 | |
| *** Guest18639 has quit IRC | 11:04 | |
| admin0 | noonedeadpunk, yes | 11:06 |
| jrosser | tacco: https://review.opendev.org/#/c/671762/ | 11:06 |
| *** nicolasbock has joined #openstack-ansible | 11:06 | |
| *** masterpe has quit IRC | 11:07 | |
| *** csmart has quit IRC | 11:07 | |
| *** openstackstatus has quit IRC | 11:09 | |
| *** csmart has joined #openstack-ansible | 11:16 | |
| *** alvinstarr has joined #openstack-ansible | 11:18 | |
| jrosser | noonedeadpunk: seems debian breaks everywhere | 11:19 |
| noonedeadpunk | yep:( | 11:19 |
| noonedeadpunk | I guess it's time to reach infra?) | 11:20 |
| *** masterpe has joined #openstack-ansible | 11:22 | |
| *** SmearedBeard has joined #openstack-ansible | 11:30 | |
| *** ansmith has quit IRC | 11:35 | |
| masterpe | hi, If I create an instance from a instance where the metadata hw_scsi_model: virtio-scsi and hw_disk_bus: scsi has been set in the image. The root disk get a name /dev/vda. The libvirt config have the line: <target dev='vda' bus='scsi'/> If I look into the instance. The disk is a /dev/sda | 11:37 |
| *** SmearedBeard has quit IRC | 11:38 | |
| *** rpittau is now known as rpittau|afk | 11:38 | |
| masterpe | If I attache a new volume (the volume does not have the metadata set). The libvirt has <target dev='sda' bus='scsi'/>. | 11:39 |
| noonedeadpunk | that sounds very familiar and I can probably recall some bug report regarding it | 11:40 |
| noonedeadpunk | isn't it https://bugs.launchpad.net/ubuntu/+source/horizon/+bug/1822129 ? | 11:41 |
| openstack | Launchpad bug 1822129 in horizon (Ubuntu Xenial) "[SRU] leave device name empty so that nova can determine it instead" [Medium,In progress] | 11:41 |
| noonedeadpunk | It was originaly https://bugs.launchpad.net/nova/+bug/1560965 iirc but moved to horizon | 11:42 |
| openstack | Launchpad bug 1560965 in OpenStack Dashboard (Horizon) "libvirt selects wrong root device name" [Low,Fix released] - Assigned to Eric Desrochers (slashd) | 11:42 |
| *** SmearedBeard has joined #openstack-ansible | 11:43 | |
| masterpe | noonedeadpunk: The thing is that if the instances with the extra attached volume boots up again. It trys to boot from sda and not from vda. there I think S is before the V in the alphabet and there is no boot order in libvirt. | 11:45 |
| noonedeadpunk | that's exactly what these bugs were about I guess | 11:45 |
| noonedeadpunk | or not this one... | 11:46 |
| noonedeadpunk | Have you mentioned fixes applied for your deployment? | 11:48 |
| *** lkoranda has joined #openstack-ansible | 11:52 | |
| *** ivve has joined #openstack-ansible | 11:56 | |
| ChosSimbaOne | noonedeadpunk: So i got it working with the cors changes. Thank you for the help. Now i just need to build a good dashboad :-D | 11:59 |
| noonedeadpunk | ChosSimbaOne: can wish you good luck:) | 12:00 |
| *** goldyfruit_ has quit IRC | 12:03 | |
| *** jbadiapa has joined #openstack-ansible | 12:06 | |
| *** goldyfruit_ has joined #openstack-ansible | 12:11 | |
| *** SmearedBeard has quit IRC | 12:14 | |
| *** goldyfruit_ has quit IRC | 12:17 | |
| *** ianychoi has quit IRC | 12:21 | |
| *** ianychoi has joined #openstack-ansible | 12:25 | |
| *** ansmith has joined #openstack-ansible | 12:27 | |
| *** jbadiapa has quit IRC | 12:35 | |
| *** hwoarang has quit IRC | 13:01 | |
| *** hwoarang has joined #openstack-ansible | 13:03 | |
| *** cshen has quit IRC | 13:10 | |
| *** SmearedBeard has joined #openstack-ansible | 13:10 | |
| *** ivve has quit IRC | 13:12 | |
| *** luksky has quit IRC | 13:14 | |
| *** cshen has joined #openstack-ansible | 13:17 | |
| jrosser | noonedeadpunk: i ran that debian apt install locally just to double check and it is fine http://paste.openstack.org/show/780746/ | 13:18 |
| *** jawad_axd has quit IRC | 13:27 | |
| *** jawad_axd has joined #openstack-ansible | 13:28 | |
| *** jawad_ax_ has joined #openstack-ansible | 13:30 | |
| *** jawad_axd has quit IRC | 13:33 | |
| *** jawad_ax_ has quit IRC | 13:34 | |
| *** luksky has joined #openstack-ansible | 13:42 | |
| *** sum12 has quit IRC | 13:45 | |
| *** sum12 has joined #openstack-ansible | 13:46 | |
| *** lkoranda has quit IRC | 13:47 | |
| *** zbr|lunch is now known as zbr|ruck | 13:48 | |
| *** zul has joined #openstack-ansible | 13:51 | |
| *** markvoelker has joined #openstack-ansible | 13:53 | |
| *** markvoelker has quit IRC | 13:58 | |
| *** prometheanfire has quit IRC | 13:58 | |
| *** prometheanfire has joined #openstack-ansible | 13:59 | |
| *** goldyfruit has joined #openstack-ansible | 14:09 | |
| *** gkadam has joined #openstack-ansible | 14:12 | |
| *** gkadam has quit IRC | 14:12 | |
| *** goldyfruit_ has joined #openstack-ansible | 14:15 | |
| *** goldyfruit has quit IRC | 14:17 | |
| *** BjoernT has joined #openstack-ansible | 14:26 | |
| *** markvoelker has joined #openstack-ansible | 14:26 | |
| *** openstackstatus has joined #openstack-ansible | 14:27 | |
| *** ChanServ sets mode: +v openstackstatus | 14:27 | |
| *** goldyfruit___ has joined #openstack-ansible | 14:33 | |
| *** goldyfruit_ has quit IRC | 14:35 | |
| *** luksky has quit IRC | 14:48 | |
| *** cshen has quit IRC | 15:08 | |
| *** macz has joined #openstack-ansible | 15:17 | |
| *** macz has quit IRC | 15:22 | |
| *** markvoelker has quit IRC | 15:25 | |
| *** goldyfruit_ has joined #openstack-ansible | 15:30 | |
| *** gyee has joined #openstack-ansible | 15:30 | |
| *** goldyfruit___ has quit IRC | 15:33 | |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible master: Use parallel git clone https://review.opendev.org/588372 | 15:40 |
| *** tosky_ has joined #openstack-ansible | 15:41 | |
| *** tosky is now known as Guest83058 | 15:41 | |
| *** tosky_ is now known as tosky | 15:41 | |
| *** Guest83058 has quit IRC | 15:43 | |
| *** ianychoi has quit IRC | 15:51 | |
| *** BjoernT has quit IRC | 15:53 | |
| *** ianychoi has joined #openstack-ansible | 15:54 | |
| *** chaptertoaster has quit IRC | 15:54 | |
| *** BjoernT has joined #openstack-ansible | 15:56 | |
| *** pcaruana has quit IRC | 16:10 | |
| *** SmearedBeard has quit IRC | 16:10 | |
| *** pcaruana has joined #openstack-ansible | 16:12 | |
| *** jawad_axd has joined #openstack-ansible | 16:30 | |
| *** jawad_axd has quit IRC | 16:35 | |
| *** tosky_ has joined #openstack-ansible | 16:40 | |
| *** tosky has quit IRC | 16:43 | |
| *** jawad_axd has joined #openstack-ansible | 16:44 | |
| *** tosky_ is now known as tosky | 16:45 | |
| *** SmearedBeard has joined #openstack-ansible | 16:58 | |
| *** manjeets has quit IRC | 16:59 | |
| *** jawad_axd has quit IRC | 17:10 | |
| *** SmearedBeard has quit IRC | 17:14 | |
| *** SmearedBeard has joined #openstack-ansible | 17:14 | |
| *** SmearedBeard has quit IRC | 17:18 | |
| *** BjoernT has quit IRC | 17:22 | |
| gillesMo | Hello ! I'm in a mixed OS version while upgrading to Ubuntu 18 and repos seems to build eroneous packages. While redeploying my second controller (no similar problem with the first one), keystone is reinstalled in the third controller (still Ubuntu 16) but cannot start because it search a libssl1.1.so library that is in Ubuntu 18 not 16. | 17:22 |
| gillesMo | repo-build.yml does not help. | 17:23 |
| *** luksky has joined #openstack-ansible | 17:43 | |
| *** pcaruana has quit IRC | 17:45 | |
| jrosser | maybe —limit localhost,controller2 when installing the new stuff? | 18:26 |
| jrosser | mnaser: check this out, bootstrap-ansible 5x quicker on local tests :) https://review.opendev.org/#/c/588372/ | 18:28 |
| *** BjoernT has joined #openstack-ansible | 18:44 | |
| *** BjoernT has quit IRC | 18:49 | |
| *** BjoernT has joined #openstack-ansible | 18:52 | |
| *** dave-mccowan has joined #openstack-ansible | 18:59 | |
| *** kopecmartin is now known as kopecmartin|off | 19:05 | |
| *** spsurya has quit IRC | 19:09 | |
| mnaser | jrosser: oh neat, looks like debian failures are infra related? | 19:11 |
| jrosser | yeah long discussion with them today about that | 19:12 |
| jrosser | dib needs a bit of fiddling to make the security repo come from the same place during image build and in CI runs | 19:12 |
| mnaser | jrosser: im also curious how centos fails with git missing | 19:12 |
| mnaser | yet in the job -- Failed to find required executable git in paths: /usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin | 19:12 |
| jrosser | because of your nova cleanup :) | 19:12 |
| jrosser | I have a patch to fix it | 19:13 |
| mnaser | ah darn | 19:13 |
| mnaser | i knew id break something.. | 19:13 |
| jrosser | it passes in the role test because metal sees all the packages from everywhere | 19:13 |
| mnaser | so does nova uninstall git then? | 19:13 |
| jrosser | but lxc job is a good proxy for multinode because you have to get the package installs right | 19:14 |
| jrosser | it doesn’t see it host git from in the nova container | 19:14 |
| mnaser | oh gotcha | 19:14 |
| mnaser | i guess maybe python_venv_build should install git? | 19:14 |
| mnaser | (if it sees git in the reqs anyways) | 19:15 |
| mnaser | which is always but i guess its possible for someone to pin to a pypi released thing | 19:15 |
| jrosser | venv build installs stuff if repo server is t there? toolchain etc I think | 19:16 |
| *** BjoernT has quit IRC | 19:37 | |
| jrosser | guilhermesp: just discussing the octaiva image out-of-disk-space in #openstack-lbaas | 19:42 |
| *** BjoernT has joined #openstack-ansible | 19:47 | |
| guilhermesp | oh cool jrosser let me get into that channel | 19:51 |
| guilhermesp | seems that im in the middle of the conversation :P | 19:52 |
| *** ansmith_ has joined #openstack-ansible | 19:58 | |
| *** ansmith has quit IRC | 20:00 | |
| *** dave-mccowan has quit IRC | 20:04 | |
| *** dave-mccowan has joined #openstack-ansible | 20:10 | |
| *** markvoelker has joined #openstack-ansible | 20:21 | |
| *** jtro has joined #openstack-ansible | 21:03 | |
| *** tosky_ has joined #openstack-ansible | 21:05 | |
| *** tosky has quit IRC | 21:06 | |
| *** tosky_ is now known as tosky | 21:08 | |
| *** ansmith_ has quit IRC | 21:09 | |
| *** markvoelker has quit IRC | 21:11 | |
| *** markvoelker has joined #openstack-ansible | 21:12 | |
| *** BjoernT has quit IRC | 21:30 | |
| *** BjoernT has joined #openstack-ansible | 21:35 | |
| *** markvoelker has quit IRC | 21:41 | |
| *** markvoelker has joined #openstack-ansible | 21:41 | |
| *** markvoelker has quit IRC | 21:45 | |
| *** markvoelker has joined #openstack-ansible | 21:45 | |
| *** ansmith_ has joined #openstack-ansible | 21:48 | |
| *** BjoernT has quit IRC | 21:50 | |
| *** BjoernT has joined #openstack-ansible | 21:54 | |
| *** carlosmss has quit IRC | 21:56 | |
| *** markvoelker has quit IRC | 22:17 | |
| *** BjoernT has quit IRC | 22:24 | |
| *** goldyfruit_ has quit IRC | 22:33 | |
| *** tosky has quit IRC | 23:16 | |
| *** luksky has quit IRC | 23:22 | |
| *** jtro has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!