| *** dave-mccowan has joined #openstack-kolla | 00:20 | |
| *** kplant has quit IRC | 01:10 | |
| *** guozijn has joined #openstack-kolla | 01:18 | |
| *** zhanglong has joined #openstack-kolla | 01:27 | |
| *** owalsh has joined #openstack-kolla | 01:39 | |
| *** dave-mccowan has quit IRC | 01:45 | |
| *** goldyfruit has joined #openstack-kolla | 02:23 | |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible master: doc: Start using openstackdoctheme's extlink extension https://review.opendev.org/665330 | 02:26 |
|---|---|---|
| *** goldyfruit has quit IRC | 02:42 | |
| *** guozijn has quit IRC | 02:48 | |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible master: doc: Start using openstackdoctheme's extlink extension https://review.opendev.org/665330 | 02:49 |
| *** guozijn has joined #openstack-kolla | 02:51 | |
| openstackgerrit | Zijian Guo proposed openstack/kolla-ansible master: Convert '-' to '_' in ansible facts for interface names https://review.opendev.org/623976 | 03:02 |
| *** whoami-rajat has joined #openstack-kolla | 03:12 | |
| *** guozijn has quit IRC | 03:21 | |
| *** zhanglong has quit IRC | 03:41 | |
| *** guozijn has joined #openstack-kolla | 03:46 | |
| *** janki has joined #openstack-kolla | 03:47 | |
| *** guozijn has quit IRC | 03:50 | |
| *** janki has quit IRC | 03:54 | |
| *** janki has joined #openstack-kolla | 03:55 | |
| *** janki has quit IRC | 04:13 | |
| *** janki has joined #openstack-kolla | 04:20 | |
| *** Sravan has joined #openstack-kolla | 04:34 | |
| *** Sravan has quit IRC | 04:59 | |
| *** Sravan has joined #openstack-kolla | 05:00 | |
| *** sshnaidm|off has quit IRC | 05:01 | |
| *** Sravan has quit IRC | 05:04 | |
| *** ivve has quit IRC | 05:30 | |
| *** janki has quit IRC | 05:36 | |
| *** janki has joined #openstack-kolla | 05:42 | |
| *** janki has quit IRC | 05:43 | |
| *** janki has joined #openstack-kolla | 05:44 | |
| hrw | morning | 05:47 |
| *** factor has joined #openstack-kolla | 05:48 | |
| *** janki has quit IRC | 05:49 | |
| *** janki has joined #openstack-kolla | 05:49 | |
| *** zhanglong has joined #openstack-kolla | 05:52 | |
| *** janki has quit IRC | 05:53 | |
| *** janki has joined #openstack-kolla | 05:53 | |
| *** luksky has joined #openstack-kolla | 05:55 | |
| *** Luzi has joined #openstack-kolla | 05:56 | |
| *** jbadiapa has joined #openstack-kolla | 06:07 | |
| *** cah_link has joined #openstack-kolla | 06:09 | |
| *** janki has quit IRC | 06:18 | |
| *** dpawlik has joined #openstack-kolla | 06:23 | |
| *** zhanglong has quit IRC | 06:34 | |
| *** ivve has joined #openstack-kolla | 06:46 | |
| *** gregoryo has joined #openstack-kolla | 06:52 | |
| *** zhanglong has joined #openstack-kolla | 06:54 | |
| *** rpittau|afk is now known as rpittau | 06:56 | |
| yoctozepto | morning | 07:01 |
| hrw | yoctozepto: I see that you go through project like tornado ;D just less things gets destroyed ;D | 07:01 |
| mnasiadka | morning | 07:02 |
| yoctozepto | hrw: no better way to learn your way through the code than acting like a tornado ;D (PS: hopefully none destroyed!) | 07:02 |
| hrw | mnasiadka: hej | 07:03 |
| mnasiadka | i hate mondays :) | 07:03 |
| hrw | mnasiadka: I hate PLL LOT :D | 07:04 |
| mnasiadka | hrw: landed in Olsztyn, or what? :D | 07:04 |
| hrw | mnasiadka: extra 24h in Warsaw | 07:04 |
| mnasiadka | hrw: uhh, I'm not a fan of this city ;) | 07:05 |
| yoctozepto | hrw: pff | 07:05 |
| yoctozepto | I hate them because they did not give us the promised snack (prince polo!) | 07:05 |
| hrw | mnasiadka: have to upgrade magnet from Olsztyn so would not be so bad ;D | 07:05 |
| yoctozepto | hrw: do your magnets have versions? | 07:05 |
| yoctozepto | do you keep them all in git? | 07:05 |
| hrw | yoctozepto: https://marcin.juszkiewicz.com.pl/my-collection-of-fridge-magnets/ | 07:06 |
| mnasiadka | yoctozepto: that's why all my Nordic colleagues liked to fly with LOT - they love Prince Polo ;) | 07:06 |
| yoctozepto | nice collection! | 07:06 |
| hrw | yoctozepto: I have my preferences when it comes to fridge magnets. Olszyn one was bought by a friend | 07:06 |
| yoctozepto | I see | 07:07 |
| hrw | yoctozepto: my collection includes one-of-a-kind ones too ;D | 07:07 |
| hrw | yoctozepto: top row has one from Pruszcz Gdański - I was told that this is a one of a kind prototype ;D | 07:08 |
| yoctozepto | hrw: sorry, I am not a specialist when it comes to fridge magnets, probably do not appreciate it as much as you do :D | 07:09 |
| *** Wasaac has joined #openstack-kolla | 07:58 | |
| *** Wasaac has quit IRC | 08:01 | |
| *** Wasaac has joined #openstack-kolla | 08:04 | |
| *** sshnaidm has joined #openstack-kolla | 08:08 | |
| *** ktibi has joined #openstack-kolla | 08:08 | |
| *** dougsz has joined #openstack-kolla | 08:08 | |
| *** Wasaac has quit IRC | 08:08 | |
| *** sshnaidm has quit IRC | 08:14 | |
| *** gregoryo has quit IRC | 08:15 | |
| *** sshnaidm has joined #openstack-kolla | 08:17 | |
| *** Wasaac has joined #openstack-kolla | 08:19 | |
| *** k_mouza has joined #openstack-kolla | 08:20 | |
| *** hjensas|afk is now known as hjensas | 08:34 | |
| *** Wasaac has quit IRC | 08:49 | |
| *** Wasaac_ has joined #openstack-kolla | 08:49 | |
| *** jbadiapa has quit IRC | 08:50 | |
| *** dciabrin has quit IRC | 08:51 | |
| *** dciabrin has joined #openstack-kolla | 08:51 | |
| *** pcaruana has joined #openstack-kolla | 08:52 | |
| *** luksky has quit IRC | 08:54 | |
| *** Wasaac_ has quit IRC | 08:59 | |
| *** jbadiapa has joined #openstack-kolla | 09:03 | |
| openstackgerrit | Pierre Riteau proposed openstack/kolla-ansible stable/stein: Stop duplicating Nova cells https://review.opendev.org/665618 | 09:14 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: WIP: Ensure ceph cluster is healthy after the upgrade https://review.opendev.org/665620 | 09:18 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: There is no Java 8 in Debian 'buster' https://review.opendev.org/665621 | 09:19 |
| *** luksky has joined #openstack-kolla | 09:27 | |
| *** Wasaac has joined #openstack-kolla | 09:29 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add kolla-build-debian-source-arm64 experimental job https://review.opendev.org/557659 | 09:32 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: WIP: Ensure ceph cluster is healthy after the upgrade https://review.opendev.org/665620 | 09:33 |
| *** lemko has joined #openstack-kolla | 09:35 | |
| *** gfidente has joined #openstack-kolla | 09:42 | |
| *** harsha has quit IRC | 09:48 | |
| *** jbadiapa has quit IRC | 09:54 | |
| *** gfidente has quit IRC | 09:56 | |
| *** k_mouza has quit IRC | 09:57 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Run nova api_db sync before db sync https://review.opendev.org/665634 | 10:04 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Run nova api_db sync before db sync https://review.opendev.org/665634 | 10:07 |
| *** Wasaac has quit IRC | 10:07 | |
| *** jbadiapa has joined #openstack-kolla | 10:07 | |
| *** Wasaac has joined #openstack-kolla | 10:08 | |
| *** k_mouza has joined #openstack-kolla | 10:13 | |
| *** ktibi has quit IRC | 10:30 | |
| *** harsha has joined #openstack-kolla | 10:32 | |
| *** k_mouza has quit IRC | 10:35 | |
| *** k_mouza has joined #openstack-kolla | 10:36 | |
| *** gfidente has joined #openstack-kolla | 10:57 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Run nova-status upgrade check after DB sync and migrations https://review.opendev.org/665643 | 11:02 |
| harsha | hello | 11:10 |
| *** zhanglong has quit IRC | 11:11 | |
| *** kplant has joined #openstack-kolla | 11:13 | |
| *** schwicht has joined #openstack-kolla | 11:28 | |
| *** Wasaac has quit IRC | 11:30 | |
| harsha | tried a fresh install, if i put enable_haproxy="yes", i get this error -> http://paste.openstack.org/show/753077/, if i put enable_haproxy="no", i get this error -> http://paste.openstack.org/show/753078/. | 11:34 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: There is no Java 8 in Debian 'buster' https://review.opendev.org/665621 | 11:37 |
| harsha | multinode -> http://paste.openstack.org/show/753079/, control node "ip a" -> http://paste.openstack.org/show/753080/, globals.yml -> http://paste.openstack.org/show/753081/ | 11:38 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: WIP: Ensure ceph cluster is healthy after the upgrade https://review.opendev.org/665620 | 11:46 |
| *** zhanglong has joined #openstack-kolla | 11:48 | |
| harsha | can check mariadb using mysql -> http://paste.openstack.org/show/753083/ | 11:49 |
| harsha | ping yoctozepto, tried a fresh install | 11:51 |
| yoctozepto | harsha: what version of kolla-ansible are you using? | 11:54 |
| yoctozepto | and of ansible | 11:54 |
| hrw | yoctozepto: https://review.opendev.org/#/c/664252/ - please address it | 11:58 |
| *** Wasaac has joined #openstack-kolla | 12:01 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Change port of docker registry https://review.opendev.org/586392 | 12:02 |
| *** edleafe has joined #openstack-kolla | 12:02 | |
| yoctozepto | hrw: yes, sir :P | 12:02 |
| hrw | yoctozepto: the golden rule is "new stuff -> master and then -> stable/" | 12:04 |
| hrw | yoctozepto: changes going only to stable/ usually are "use versioned tarballs instead of master/HEAD ones" etc | 12:04 |
| hrw | yoctozepto: sometimes it is needed to squash two/more changes into one backport. but normally it is just cherry-picking | 12:05 |
| yoctozepto | hrw: yeah, I should have squashed your change with mine, to avoid percona headaches | 12:06 |
| *** ktibi has joined #openstack-kolla | 12:06 | |
| *** Wasaac has quit IRC | 12:06 | |
| yoctozepto | seriously, what's wrong with their servers :S | 12:06 |
| hrw | one thing to consider dropping :D | 12:09 |
| *** ktibi has quit IRC | 12:11 | |
| yoctozepto | hrw: the blueprint is there | 12:12 |
| *** ktibi has joined #openstack-kolla | 12:13 | |
| yoctozepto | btw, what's the actual process in kolla with specs, blueprints and stuff? | 12:13 |
| *** zhanglong has quit IRC | 12:17 | |
| *** zhanglong has joined #openstack-kolla | 12:19 | |
| *** Wasaac has joined #openstack-kolla | 12:23 | |
| *** Wasaac has quit IRC | 12:32 | |
| *** Wasaac_ has joined #openstack-kolla | 12:32 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Add centos_yum_repos_to_disable overridable in base https://review.opendev.org/665657 | 12:37 |
| mgoddard | yoctozepto: we don't generally require specs for a feature, only a blueprint. Specs can be useful to discuss design details for bigger things | 12:42 |
| *** dave-mccowan has joined #openstack-kolla | 12:42 | |
| hrw | yoctozepto: 'Add a way to disable centos repos in controlable way' sounds better | 12:45 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Restart all nova services after upgrade https://review.opendev.org/665660 | 12:56 |
| *** hamzaachi has joined #openstack-kolla | 12:57 | |
| hrw | https://pastebin.com/0TBAZ50v - which wheels are build where. I will send some patches... | 12:58 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/stein: Add centos_yum_repos_to_disable overridable in base https://review.opendev.org/664252 | 12:58 |
| hrw | yoctozepto: first get stuff merged in master. then use 'cherry-pick' button in gerrit to propose backport | 13:01 |
| hrw | yoctozepto: do not develop feature for >1 branches at same time (unless really needed) | 13:01 |
| hrw | yoctozepto: so for 664252 do commit message edit, get it into master, then to stein then add patch which uses it in stein | 13:02 |
| *** schwicht has quit IRC | 13:09 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: monasca: add 'confluent-kafka' to base https://review.opendev.org/665662 | 13:12 |
| *** lennyb has joined #openstack-kolla | 13:13 | |
| yoctozepto | > <hrw> yoctozepto: 'Add a way to disable centos repos in controlable way' sounds better | 13:19 |
| yoctozepto | ah, marketing! | 13:19 |
| yoctozepto | <hrw> yoctozepto: first get stuff merged in master. then use 'cherry-pick' button in gerrit to propose backport | 13:19 |
| hrw | yoctozepto: not marketing. sensible line | 13:19 |
| *** schwicht has joined #openstack-kolla | 13:19 | |
| yoctozepto | ok :-) | 13:19 |
| hrw | yoctozepto: if I have to read code to findout what patch does then commit message is shit. | 13:20 |
| yoctozepto | :-( | 13:20 |
| hrw | yoctozepto: if I have to read whole commit message + code then first line is ubershit | 13:20 |
| yoctozepto | :-(( | 13:20 |
| hrw | yoctozepto: "git log --oneline" should inform | 13:20 |
| *** goldyfruit has joined #openstack-kolla | 13:24 | |
| *** pcaruana has quit IRC | 13:26 | |
| *** pcaruana|afk| has joined #openstack-kolla | 13:26 | |
| *** ktibi has quit IRC | 13:26 | |
| *** ktibi has joined #openstack-kolla | 13:27 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: openstack-base: add several popular python modules https://review.opendev.org/665668 | 13:29 |
| hrw | now rebuilding all images to check does it improve situation | 13:30 |
| mgoddard | hrw: play nicely :p | 13:33 |
| yoctozepto | hrw: hopefully does not increase the risk of getting too-new deps which have to be downgraded :/ | 13:33 |
| hrw | mgoddard: I think that Polish people are used to more direct talk | 13:33 |
| hrw | yoctozepto: it adds images popular in other images | 13:34 |
| mgoddard | hrw: yoctozepto is still here, so maybe you are right :) | 13:34 |
| hrw | https://pastebin.com/raw/0TBAZ50v shows which wheel is built in which image | 13:34 |
| yoctozepto | hrw: I mean a situation like the project does blah<4.0.0 and the base image already downloaded blah==4.1.2 | 13:35 |
| hrw | yoctozepto: we have such ones here and there already | 13:36 |
| hrw | "project XY wanted blablab <1.2 but 1.4 was already installed" | 13:36 |
| yoctozepto | hrw: are we dealing with this? | 13:37 |
| hrw | yoctozepto: iirc our pip calls use upper-contraints from requirements or sth like that | 13:38 |
| yoctozepto | hrw: good then | 13:38 |
| hrw | but may be wrong | 13:38 |
| yoctozepto | hrw: bad then | 13:38 |
| hrw | INFO:kolla.common.utils.openstack-base:Step 8/8 : RUN pip --no-cache-dir install --upgrade -c /requirements/upper-constraints.txt | 13:38 |
| yoctozepto | so we should be fine | 13:39 |
| hrw | now 714 images to remove to get clean rebuild | 13:39 |
| mnasiadka | Two polish guys and so much activity on the channel | 13:43 |
| hrw | mnasiadka: you do not count? | 13:43 |
| * hrw -> food | 13:44 | |
| mnasiadka | hrw: I’m rather quiet today ;) | 13:44 |
| mnasiadka | Having fun with molecule | 13:44 |
| *** pcaruana has joined #openstack-kolla | 13:48 | |
| *** mchlumsky has joined #openstack-kolla | 13:48 | |
| *** pcaruana|afk| has quit IRC | 13:51 | |
| *** ktibi has quit IRC | 13:56 | |
| *** Luzi has quit IRC | 14:03 | |
| *** harsha_ has joined #openstack-kolla | 14:03 | |
| harsha_ | ansible version is 2.5.1, how do i check kolla-ansible version? (i am using master branch from git) , ping yoctozepto | 14:05 |
| mnasiadka | harsha_: pip freeze | grep kolla-ansible | 14:06 |
| harsha_ | it says 7.1.0 but it might be from stable version, at the very beginning i tried stable one | 14:11 |
| harsha_ | mnasiadka: any idea what i could do resolve the error? (i have posted it previously) | 14:15 |
| mnasiadka | harsha_: try ansible 2.5.5 or newer - like 2.7, seems like you're hitting a bug in ansible | 14:18 |
| *** ktibi has joined #openstack-kolla | 14:18 | |
| mnasiadka | harsha_: https://github.com/ansible/ansible/issues/23609 | 14:19 |
| *** dpawlik has quit IRC | 14:19 | |
| harsha_ | hmm ill try that | 14:21 |
| *** nathharp has joined #openstack-kolla | 14:31 | |
| *** JamesBenson has joined #openstack-kolla | 14:33 | |
| *** cah_link has quit IRC | 14:35 | |
| *** Wasaac_ has quit IRC | 14:35 | |
| *** Wasaac has joined #openstack-kolla | 14:35 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: There is no Java 8 in Debian 'buster' https://review.opendev.org/665621 | 14:40 |
| harsha_ | mnasiadka: how do i install the newer versions? | 14:40 |
| harsha_ | via apt it says 2.5.1 is the latest | 14:40 |
| hrw | mnasiadka: https://review.opendev.org/662429 - can you? | 14:40 |
| mnasiadka | harsha_: pip install -U ansible==2.5.5 (or without the ==2.5.5 and you get newest) | 14:40 |
| mnasiadka | hrw: sure :) | 14:41 |
| mnasiadka | harsha_: or add an apt repo with fresh ansible | 14:41 |
| mnasiadka | harsha_: if you want to use pip - remove the apt package first | 14:42 |
| hrw | mnasiadka: https://review.opendev.org/586392 maybe too? | 14:42 |
| hrw | :D | 14:42 |
| harsha_ | well, got 2.8.1 lol | 14:42 |
| mnasiadka | hrw: yeah, I don't like this ;) | 14:45 |
| *** cah_link has joined #openstack-kolla | 14:46 | |
| *** rgogunskiy has joined #openstack-kolla | 14:47 | |
| *** pcaruana has quit IRC | 14:47 | |
| harsha_ | mnasiadka: it passwd through that step! | 14:53 |
| mnasiadka | harsha_: let's see where it fails now ;) | 14:54 |
| harsha_ | it did | 14:54 |
| harsha_ | pasting error msg | 14:54 |
| harsha_ | mnasiadka: http://paste.openstack.org/show/753094/ | 14:54 |
| mnasiadka | harsha_: docker logs haproxy please | 14:55 |
| openstackgerrit | Merged openstack/kolla stable/rocky: Handle role properly when it is changed in kolla_keystone_user module https://review.opendev.org/664979 | 14:55 |
| openstackgerrit | Merged openstack/kolla stable/queens: Handle role properly when it is changed in kolla_keystone_user module https://review.opendev.org/664980 | 14:55 |
| *** ktibi has quit IRC | 14:55 | |
| *** ktibi has joined #openstack-kolla | 14:56 | |
| harsha_ | mnasiadka: i think its stuck in restart | 14:59 |
| harsha_ | http://paste.openstack.org/show/753096/ | 14:59 |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible master: doc: Start using openstackdoctheme's extlink extension https://review.opendev.org/665330 | 15:02 |
| *** gfidente has quit IRC | 15:03 | |
| *** ivve has quit IRC | 15:05 | |
| harsha_ | mnasiadka: i can't even run the docker, it says the container is restarting | 15:07 |
| mnasiadka | harsha_: you should be able to find something either in /var/log/kolla/haproxy/ or in docker logs haproxy | 15:15 |
| *** luksky has quit IRC | 15:20 | |
| harsha_ | mnasiadka: http://paste.openstack.org/show/753098/ | 15:21 |
| mnasiadka | harsha_: interesting :) | 15:21 |
| harsha_ | the whole thing didnt get copied | 15:23 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: openstack-base: add several popular python modules https://review.opendev.org/665668 | 15:24 |
| harsha_ | mnasiadka: i think paste.openstack.org has a limit | 15:27 |
| harsha_ | mnasiadka: https://pastebin.com/vrAM2GuU | 15:29 |
| *** pcaruana has joined #openstack-kolla | 15:40 | |
| *** ktibi has quit IRC | 15:40 | |
| *** rgogunskiy has quit IRC | 15:46 | |
| *** Wasaac has quit IRC | 15:47 | |
| openstackgerrit | Merged openstack/kolla stable/stein: [neutron] Do not migrate all subprojects https://review.opendev.org/665532 | 15:49 |
| *** cah_link has quit IRC | 15:51 | |
| *** Wasaac has joined #openstack-kolla | 15:51 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Simplify handler conditionals https://review.opendev.org/647699 | 15:54 |
| *** Wasaac has quit IRC | 15:55 | |
| mgoddard | Would really love it if we could merge this: https://review.opendev.org/#/c/647699/ | 15:57 |
| hrw | mnasiadka: ^^ you have ansible knowledge | 16:00 |
| mnasiadka | looks like a nice monster | 16:00 |
| hrw | out of my ansible knowledge ;( | 16:04 |
| mnasiadka | but I always was interested why we do those checks, if Ansible doesn't even run those playbooks if those checks are not fulfilled ;) | 16:04 |
| mnasiadka | looks good to me | 16:08 |
| openstackgerrit | Merged openstack/kolla master: grafana/aarch64: use package from upstream repositories https://review.opendev.org/662429 | 16:09 |
| hrw | ok, now time for https://review.opendev.org/665668 :D once stuff gets installed only in openstack-base then build time should be shorter so finally CI jobs for aarch64 may have a chance | 16:12 |
| hrw | tomorrow EPEL mirror will get new sync which should result in aarch64 being present. then CentOS/aarch64 should work (or timeout) | 16:12 |
| *** schwicht has quit IRC | 16:14 | |
| *** lemko has quit IRC | 16:15 | |
| harsha_ | any solution to this error> -> http://paste.openstack.org/show/753106/ | 16:15 |
| klindgren | yoctozepto, No hard feelings here. I can rebase it ontop of Marks commit to remove the handler conditionals. I also see what you are talking about re: the conditionals for if kolla_action. I had spot checked a few and didn't see that. But all of the include_tasks, have the conditional on it. So I think I will just remove the conditional check across all the check-containers, for kolla-action != config. | 16:22 |
| *** Wasaac has joined #openstack-kolla | 16:25 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove nova-consoleauth https://review.opendev.org/661427 | 16:28 |
| *** hamzaachi has quit IRC | 16:29 | |
| *** Wasaac has quit IRC | 16:29 | |
| *** whoami-rajat has quit IRC | 16:29 | |
| *** rpittau is now known as rpittau|afk | 16:31 | |
| yoctozepto | klindgren: thanks, truth is I didn't know mgoddard already thought of it and did :D | 16:34 |
| yoctozepto | though it feels nice to be right 8-) | 16:34 |
| *** nathharp_ has joined #openstack-kolla | 16:35 | |
| klindgren | he did it because I brought it up, when I first went to do the change. | 16:35 |
| klindgren | I was like WTAF, in this channel. :-D | 16:35 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: base: do not use non-free repositories for Ubuntu https://review.opendev.org/665724 | 16:36 |
| yoctozepto | klindgren: hahaha, I see | 16:36 |
| hrw | mgoddard, mnasiadka: drop everything and +2 ^^ | 16:36 |
| mgoddard | klindgren: actually it was your ex-colleague (I think) rm_work who made me realise how we don't need them :) | 16:37 |
| hrw | INFO:kolla.common.utils.base:1mFailed to fetch http://mirror.iad.rax.opendev.org/ubuntu/dists/bionic-updates/multiverse/binary-amd64/Packages 404 Not Found [IP: 104.130.4.160 80]m | 16:37 |
| yoctozepto | hrw: +1 | 16:37 |
| yoctozepto | had it on my todo | 16:37 |
| hrw | we use repos which are not there | 16:37 |
| *** nathharp has quit IRC | 16:37 | |
| klindgren | ah right, I just re-brought it up. | 16:37 |
| yoctozepto | guys, it's great it's getting sorted out, no matter who brought it up | 16:38 |
| mgoddard | klindgren: one thing to bear in mind if you rebase is that the notify must now always call the correct handler - no more notifying multiple handlers | 16:38 |
| harsha_ | btw i got this error after i did a "git pull" -> http://paste.openstack.org/show/753106/ | 16:38 |
| mgoddard | klindgren: I can't remember if your change had any of those left | 16:39 |
| *** nathharp_ has quit IRC | 16:39 | |
| klindgren | I am pretty sure I changed it to do notify: restart {{ item }} container | 16:39 |
| klindgren | iirc | 16:39 |
| klindgren | on all of them | 16:39 |
| mgoddard | hrw: will see what zuul thinks | 16:39 |
| klindgren | as that was an ask earlier on | 16:39 |
| mgoddard | klindgren: cool | 16:39 |
| hrw | mgoddard: ok | 16:39 |
| klindgren | I can recheck that though. | 16:40 |
| *** eandersson has joined #openstack-kolla | 16:42 | |
| *** igordc has joined #openstack-kolla | 16:44 | |
| *** k_mouza has quit IRC | 16:47 | |
| *** whoami-rajat has joined #openstack-kolla | 16:48 | |
| *** schwicht has joined #openstack-kolla | 16:49 | |
| yoctozepto | is there an ansible style guide that we stick to? | 16:50 |
| yoctozepto | like the casing of booleans | 16:50 |
| yoctozepto | we have both True and true, false and False | 16:50 |
| yoctozepto | or not yet the stage we care about it? :P | 16:50 |
| yoctozepto | mgoddard, mnasiadka, hrw, klindgren ^ | 16:51 |
| klindgren | we also have 'yes' 'no' | 16:51 |
| mgoddard | yoctozepto: there is a .yamllint file in the repo | 16:51 |
| klindgren | :-/ | 16:51 |
| mgoddard | although it's wide open | 16:51 |
| mgoddard | I personally tend to go for true/false in code for booleans | 16:52 |
| mgoddard | but we should accept any of the above for user input/config, by passing everything through a bool filter | 16:52 |
| klindgren | eg: https://github.com/openstack/kolla-ansible/blob/master/etc/kolla/globals.yml#L117 | 16:53 |
| mgoddard | yeah | 16:54 |
| mgoddard | user config often uses yes/no | 16:54 |
| yoctozepto | yeah, user input sure | 16:54 |
| yoctozepto | but for the code | 16:54 |
| mgoddard | true/false | 16:54 |
| yoctozepto | +1 | 16:54 |
| mgoddard | but it's not enforced | 16:55 |
| yoctozepto | keeping hand off the shift key | 16:55 |
| yoctozepto | yeah, definitely saw that! | 16:55 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Make cron jobs generator command task in keystone non-changing https://review.opendev.org/665533 | 16:56 |
| *** dougsz has quit IRC | 16:57 | |
| klindgren | So anyone have some ideas re: making kolla-ansible more performant? and using less memory. We have acloud now with 400+ HV's and twoards the end of deploy containers ansible-playbook is consuming 2.6GB of ram per fork. | 16:58 |
| *** ivve has joined #openstack-kolla | 16:58 | |
| klindgren | Kinda want to brainstorm some ideas to try some stuff out, as 400+HV's is taking ~5 hours to do deployments, where nothing changes. | 16:59 |
| *** Sravan has joined #openstack-kolla | 16:59 | |
| klindgren | Most of the time is common, nova, neutron. And most of the time is spent doing config generation, in our runs. | 17:00 |
| yoctozepto | klindgren: as far as mem is concerned, I believe we might be collecting way too many vars/facts | 17:01 |
| klindgren | yes | 17:02 |
| yoctozepto | as for the performance itself | 17:02 |
| klindgren | but also I think I know of some ansible bugs. like include_tasks uses more memory than import_tasks | 17:02 |
| yoctozepto | you tried pipelining and adjusting parallelization? | 17:03 |
| yoctozepto | that's probably not a bug, import is static | 17:03 |
| yoctozepto | it's like pasting stuff here and there | 17:03 |
| *** harsha_ has quit IRC | 17:03 | |
| yoctozepto | and include is like recursion | 17:03 |
| klindgren | well something from openshift, saw like 900mb reduction by converting to import vs's include. | 17:04 |
| *** zbr|ooo has quit IRC | 17:04 | |
| yoctozepto | yeah, could be, depending on the number of hosts in inventory | 17:04 |
| klindgren | they went from 1GB per host, to 100mb per host in inventory - or something similar. | 17:04 |
| yoctozepto | per host? oh my | 17:05 |
| klindgren | yea | 17:05 |
| yoctozepto | I thought of a few megs per host | 17:05 |
| yoctozepto | clearly, not all includes can be imports | 17:05 |
| yoctozepto | but some probably can | 17:05 |
| *** k_mouza has joined #openstack-kolla | 17:06 | |
| klindgren | We have done pipelining, running 40 forks, we tried running mitogen for ansible as well | 17:06 |
| *** zbr has joined #openstack-kolla | 17:06 | |
| klindgren | nothing has really helped making things faster. | 17:06 |
| *** zbr is now known as zbr|ruck | 17:06 | |
| yoctozepto | klindgren: how does mitogen work? | 17:08 |
| yoctozepto | their pages looks as if they were doing magic, nothing concrete | 17:08 |
| klindgren | like functionally, or pratically? | 17:09 |
| klindgren | :-D | 17:09 |
| yoctozepto | klindgren: tell me what you know :D | 17:10 |
| *** k_mouza has quit IRC | 17:10 | |
| JamesBenson | Theoretical question: If I issue a mariadb recovery would it be more effective with a newer version of kolla-ansible? or are they openstack dependent? | 17:11 |
| klindgren | I saw 0 changes with it. I assume because most of the time spent is config gen and evaluating conditionals, which appear to be done in the main ansible-playbook process. Also, they don't do whatever blackmagic with userdefined modules | 17:11 |
| klindgren | which against screws everything up due to the merge_config tasks. | 17:11 |
| klindgren | We run master-kolla ansible (well about 30 days ago master)to deploy a pike cloud. | 17:12 |
| klindgren | JamesBenson, but honestly I wouldn't recommended trying it on a production cloud | 17:13 |
| klindgren | I would focus on manually fixing mariadb | 17:13 |
| klindgren | to get your deployment working again | 17:13 |
| hrw | JamesBenson: official way is: use same version of kolla-ansible as your kolla images | 17:14 |
| JamesBenson | yeah, well 2 of the nodes seem to do fine with the recovery, the master though seems to forget that its master throughout the process. | 17:14 |
| hrw | JamesBenson: we had serious mysql/mariadb changes in one release ending in all failing when used older kolla with newer kolla-ansible. | 17:14 |
| JamesBenson | okay | 17:15 |
| JamesBenson | so I'll keep it the same then ;-) | 17:15 |
| *** Sravan has quit IRC | 17:19 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:28 | |
| yoctozepto | klindgren: thanks for insights | 17:32 |
| *** calbers has quit IRC | 17:32 | |
| *** Wasaac has joined #openstack-kolla | 17:33 | |
| *** Sravan has joined #openstack-kolla | 17:33 | |
| JamesBenson | any reason on why mariadb master container is running, but port not showing success? | 17:34 |
| *** Sravan has quit IRC | 17:34 | |
| *** Sravan has joined #openstack-kolla | 17:34 | |
| *** calbers has joined #openstack-kolla | 17:35 | |
| *** zhanglong has quit IRC | 17:35 | |
| mgoddard | klindgren: I have some ideas. Would definitely like to be involved in brainstorming as we have a very big deployment coming up | 17:35 |
| yoctozepto | JamesBenson: why do you mean by "port not showing success" | 17:35 |
| klindgren | mgoddard, Should I startup an etherpad? | 17:36 |
| yoctozepto | what* | 17:36 |
| JamesBenson | ansible state, https://github.com/openstack/kolla-ansible/blob/d93c604d7af03e8b096faea8acaaaef718df2cd5/ansible/roles/mariadb/handlers/main.yml#L134 | 17:36 |
| mgoddard | klindgren: for nova & neutron, I'd like to look at splitting out control from compute into two plays. That way we could avoid a lot of skips | 17:36 |
| JamesBenson | this is where mariadb recovery fails | 17:36 |
| mgoddard | klindgren: I also have a plan for removing all the skips entirely using jinja filters | 17:37 |
| mgoddard | klindgren: please do start an etherpad & share around | 17:37 |
| mgoddard | klindgren: maybe we should arrange a call with interested parties? | 17:37 |
| mgoddard | klindgren: certainly interested to hear what you're hitting | 17:37 |
| *** Wasaac has quit IRC | 17:39 | |
| yoctozepto | JamesBenson: have you tried connecting to it manually (i.e. by yourself) | 17:39 |
| JamesBenson | not sure the process | 17:40 |
| hrw | bye | 17:42 |
| klindgren | JamesBenson, wait_for, literally afaik, just does effectively a telnet against the host and port | 17:46 |
| klindgren | and will wait for 60 seconds for mysql to return stuff, or 1 second for the three-way handshack to happen. | 17:47 |
| klindgren | and it will retry 10 times with a 6 second delay inbetween | 17:47 |
| klindgren | handshake* | 17:47 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix placement log perms in config.json https://review.opendev.org/664631 | 17:48 |
| *** Sravan has quit IRC | 17:50 | |
| JamesBenson | yeah, ... just not sure why the commands aren't working... why the container isn't picking up. | 17:50 |
| *** Sravan has joined #openstack-kolla | 17:52 | |
| *** Sravan has quit IRC | 17:56 | |
| yoctozepto | > <JamesBenson> not sure the process | 17:56 |
| yoctozepto | mariadb? | 17:56 |
| yoctozepto | :D | 17:56 |
| yoctozepto | the container | 17:56 |
| JamesBenson | yeah | 17:56 |
| yoctozepto | docker ps | 17:57 |
| yoctozepto | inspect mariadb | 17:57 |
| JamesBenson | yoctozepto: what am I looking for exactly? The logs are stating that 0/3 have joined the cluster... | 17:58 |
| *** vishwanathj has joined #openstack-kolla | 17:58 | |
| JamesBenson | I've been tailing the logs | 17:58 |
| yoctozepto | > <JamesBenson> yoctozepto: what am I looking for exactly | 18:00 |
| yoctozepto | wish I had known! | 18:00 |
| yoctozepto | hmm | 18:00 |
| yoctozepto | 0/3 | 18:00 |
| yoctozepto | then 3 are missing | 18:00 |
| yoctozepto | there is your clue :D | 18:00 |
| JamesBenson | ...funny guy.... | 18:01 |
| yoctozepto | galera must have failed to reconverge | 18:01 |
| yoctozepto | you should have slaves running somewhere too | 18:01 |
| yoctozepto | could check those too | 18:01 |
| JamesBenson | yeah, I'm tailing all of them... | 18:01 |
| *** Wasaac has joined #openstack-kolla | 18:03 | |
| JamesBenson | so sometimes it passes that step, others it fails on "mariadb : Creating haproxy mysql user" | 18:04 |
| JamesBenson | with " 1047, 'WSREP has not yet prepared node for application use'" | 18:04 |
| yoctozepto | this one is a known problem, probably some race condition | 18:06 |
| JamesBenson | should I put in a wait then? | 18:06 |
| *** Sravan has joined #openstack-kolla | 18:07 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/ocata: Disable publishing images on Ocata https://review.opendev.org/665742 | 18:07 |
| yoctozepto | so do they converge finally? | 18:07 |
| klindgren | mgoddard - https://etherpad.openstack.org/p/kolla-ansible-speed-ups | 18:07 |
| yoctozepto | is mariadb ok after a while? | 18:07 |
| vishwanathj | I came across this link that had info on how to enable central logging using EFK https://github.com/inc0/kolla-ansible-workshop/blob/master/troubleshooting.md .... I am looking next to get info(best practices)on how to plan for storage requirements for logs....would be great if you can point me to such documents if one exists....thanks | 18:08 |
| JamesBenson | first time after many MANY retries! | 18:08 |
| mgoddard | vishwanathj: dougsz is knowledgeable about this, but he is not here currently. He is in UK timezone | 18:09 |
| vishwanathj | mgoddard thanks, will reach out to him UK time | 18:10 |
| yoctozepto | JamesBenson: odd, are the machines performant enough and the network connection between them reliable? (also, are we talking clean install or upgrade because I forgot) | 18:12 |
| JamesBenson | yoctozepto: dirty shutdown. | 18:12 |
| yoctozepto | ugh | 18:12 |
| JamesBenson | yes! | 18:12 |
| yoctozepto | agh | 18:12 |
| yoctozepto | egh | 18:12 |
| JamesBenson | in a big big way. | 18:12 |
| *** luksky has joined #openstack-kolla | 18:12 | |
| JamesBenson | at least things are talking now. | 18:13 |
| yoctozepto | JamesBenson: last time I said that, I was told to visit a doctor! | 18:14 |
| mgoddard | JamesBenson: we made a change recently to help with this | 18:14 |
| mgoddard | JamesBenson: https://review.opendev.org/644244 | 18:15 |
| mgoddard | JamesBenson: it was backported to rocky | 18:15 |
| JamesBenson | okay... that's good. because after this, my coworkers and I are even thinking about going to joyent or nebula or something... and we've been with openstack for a long time! | 18:15 |
| JamesBenson | awesome, except our clouds are ocata and queens. | 18:16 |
| JamesBenson | :-( | 18:16 |
| JamesBenson | any idea on how to fix this: https://snag.gy/mTX0S2.jpg | 18:17 |
| JamesBenson | we have ceph, and it's in a healthy state... | 18:18 |
| mgoddard | JamesBenson: sorry, nope | 18:23 |
| *** Sravan has quit IRC | 18:25 | |
| yoctozepto | JamesBenson: new vm spun? maybe the image in glance got corrupted before? | 18:25 |
| *** Wasaac has quit IRC | 18:26 | |
| JamesBenson | yeah, rebuilding the object map... | 18:26 |
| *** Sravan has joined #openstack-kolla | 18:26 | |
| *** dpawlik has joined #openstack-kolla | 18:29 | |
| *** niceplace has quit IRC | 18:30 | |
| mgoddard | night all o/ | 18:31 |
| goldyfruit | o/ | 18:31 |
| *** niceplace has joined #openstack-kolla | 18:31 | |
| *** Wasaac has joined #openstack-kolla | 18:33 | |
| yoctozepto | o/ | 18:34 |
| *** Sravan has quit IRC | 18:38 | |
| *** pcaruana has quit IRC | 18:44 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Add a variable to configure repos disabled in base CentOS image https://review.opendev.org/665657 | 18:46 |
| *** Sravan has joined #openstack-kolla | 18:49 | |
| *** Wasaac has quit IRC | 18:52 | |
| *** vishwanathj has quit IRC | 18:52 | |
| *** dave-mccowan has quit IRC | 18:54 | |
| *** Sravan has quit IRC | 18:59 | |
| *** Sravan has joined #openstack-kolla | 19:03 | |
| *** dave-mccowan has joined #openstack-kolla | 19:05 | |
| *** dave-mccowan has quit IRC | 19:14 | |
| *** Wasaac has joined #openstack-kolla | 19:15 | |
| *** pcaruana has joined #openstack-kolla | 19:17 | |
| *** pcaruana has quit IRC | 19:27 | |
| *** Wasaac has quit IRC | 19:38 | |
| *** Wasaac has joined #openstack-kolla | 19:38 | |
| *** kplant has quit IRC | 19:39 | |
| *** Sravan has quit IRC | 19:43 | |
| mnasiadka | JamesBenson: are you sure there are no locks on rbds? | 19:46 |
| *** hamzaachi has joined #openstack-kolla | 20:17 | |
| *** whoami-rajat has quit IRC | 20:19 | |
| *** Wasaac has quit IRC | 20:37 | |
| *** Wasaac has joined #openstack-kolla | 20:39 | |
| JamesBenson | I was able to restore using: rbd object-map rebuild vms/2c28ee79-a39a-48c6-a4c3-43c05c1bbdfe_disk | 20:45 |
| *** dpawlik has quit IRC | 20:50 | |
| *** Wasaac has quit IRC | 20:53 | |
| *** Wasaac has joined #openstack-kolla | 20:53 | |
| *** hamzaachi has quit IRC | 21:05 | |
| *** Sravan has joined #openstack-kolla | 21:08 | |
| *** Wasaac has quit IRC | 21:22 | |
| *** hamzaachi has joined #openstack-kolla | 21:26 | |
| *** Sravan has quit IRC | 21:27 | |
| rm_work | mgoddard / klindgren ah hey :D yeah also we will be switching to k-a shortly from our internal tooling, so probably I'll be back working on stuff in a bit | 21:28 |
| rm_work | (am at Verizon now) | 21:28 |
| *** Sravan has joined #openstack-kolla | 21:29 | |
| rm_work | jroll is working on that currently (you may be aware) | 21:29 |
| klindgren | great | 21:29 |
| klindgren | so um yea | 21:29 |
| klindgren | https://etherpad.openstack.org/p/kolla-ansible-speed-ups | 21:29 |
| * jroll ducks | 21:30 | |
| klindgren | any ideas people have to try out to make things faster. That would be great. | 21:30 |
| *** Sravan has quit IRC | 21:34 | |
| *** mchlumsky has quit IRC | 21:43 | |
| *** Sravan has joined #openstack-kolla | 21:47 | |
| *** Sravan has joined #openstack-kolla | 21:52 | |
| *** Wasaac has joined #openstack-kolla | 21:52 | |
| *** lemko has joined #openstack-kolla | 21:52 | |
| *** Sravan has quit IRC | 21:57 | |
| *** Wasaac has quit IRC | 21:57 | |
| openstackgerrit | Kris Lindgren proposed openstack/kolla-ansible master: Add a job that *only* deploys updated containers https://review.opendev.org/645966 | 21:57 |
| *** Sravan has joined #openstack-kolla | 21:57 | |
| *** JamesBenson has quit IRC | 21:59 | |
| *** JamesBenson has joined #openstack-kolla | 22:00 | |
| *** hamzaachi has quit IRC | 22:00 | |
| openstackgerrit | Kris Lindgren proposed openstack/kolla-ansible master: Add a job that *only* deploys updated containers https://review.opendev.org/645966 | 22:02 |
| *** JamesBenson has quit IRC | 22:04 | |
| *** luksky has quit IRC | 22:08 | |
| *** ChanServ has quit IRC | 22:09 | |
| *** ChanServ has joined #openstack-kolla | 22:14 | |
| *** niven.freenode.net sets mode: +o ChanServ | 22:14 | |
| *** Sravan has quit IRC | 22:15 | |
| *** luksky has joined #openstack-kolla | 22:21 | |
| *** goldyfruit has quit IRC | 22:27 | |
| *** Sravan has joined #openstack-kolla | 22:31 | |
| *** luksky has quit IRC | 22:39 | |
| *** vishwanathj has joined #openstack-kolla | 22:44 | |
| vishwanathj | I am looking for instructions that lists steps to create a read-only user that can login to the Horizon UI with read only permissions....any pointers to such a link would be appreciated | 22:56 |
| *** vishwanathj has quit IRC | 23:23 | |
| *** vishwanathj has joined #openstack-kolla | 23:24 | |
| *** vishwanathj has quit IRC | 23:28 | |
| *** goldyfruit has joined #openstack-kolla | 23:43 | |
| *** Sravan has quit IRC | 23:54 | |
| *** Sravan has joined #openstack-kolla | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
