| *** dwilde has joined #openstack-ansible | 00:13 | |
| *** gshippey has quit IRC | 00:16 | |
| *** macz_ has quit IRC | 00:22 | |
| *** tosky has quit IRC | 00:36 | |
| *** d34dh0r53 has quit IRC | 00:50 | |
| *** dwilde has quit IRC | 00:50 | |
| *** dwilde has joined #openstack-ansible | 00:53 | |
| *** d34dh0r53 has joined #openstack-ansible | 00:54 | |
| *** cloudnull has quit IRC | 01:29 | |
| *** maharg101 has joined #openstack-ansible | 01:30 | |
| *** maharg101 has quit IRC | 01:35 | |
| *** dwilde has quit IRC | 01:48 | |
| *** miloa has joined #openstack-ansible | 05:32 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:35 | |
| *** CeeMac has quit IRC | 05:53 | |
| *** pto has joined #openstack-ansible | 06:29 | |
| *** pto has joined #openstack-ansible | 06:29 | |
| *** gyee has quit IRC | 06:31 | |
| *** akahat|rover is now known as akahat|lunch | 07:45 | |
| *** rpittau|afk is now known as rpittau | 07:47 | |
| *** andrewbonney has joined #openstack-ansible | 08:17 | |
| *** spatel has joined #openstack-ansible | 08:17 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-rsyslog_client master: Update role for Ubuntu Focal and Centos-8 https://review.opendev.org/c/openstack/openstack-ansible-rsyslog_client/+/770731 | 08:19 |
|---|---|---|
| *** maharg101 has joined #openstack-ansible | 08:20 | |
| *** spatel has quit IRC | 08:22 | |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/767770 | 08:26 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Remove chroot connection from ssh plugin https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/737273 | 08:39 |
| *** Muran has joined #openstack-ansible | 08:47 | |
| *** tosky has joined #openstack-ansible | 08:49 | |
| noonedeadpunk | dmsimard: yeah, and we can probably fix it one day, but that day hasn't come yet:) as eventually having correct diff output by the plugin would be super nice, but will be honest - it's not on top of the list :( | 09:02 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-rsyslog_client master: Update role for Ubuntu Focal and Centos-8 https://review.opendev.org/c/openstack/openstack-ansible-rsyslog_client/+/770731 | 09:10 |
| *** macz_ has joined #openstack-ansible | 09:18 | |
| *** macz_ has quit IRC | 09:22 | |
| jrosser | this is breaking what i'm doing on the new pip resolver https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L383 | 09:31 |
| jrosser | seems that because tempest is included in upper-constraints we can't have a requirement like git+https://opendev.org/openstack/tempest@7d775d6d1c0999afa1e8444b2f6fba4a432ccc20#egg=tempest | 09:32 |
| jrosser | becasue you can't simultaneously resolve tempest===<x.y.z> from u-c and the specific git version we want | 09:33 |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-os_zun master: Reinstate voting for upgrade jobs https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/769144 | 09:45 |
| openstackgerrit | Merged openstack/ansible-role-systemd_mount master: Add option to use s3fs with role https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/767702 | 10:19 |
| openstackgerrit | Merged openstack/ansible-role-systemd_service master: Add possibility to configure systemd sockets https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/763194 | 10:21 |
| openstackgerrit | Merged openstack/ansible-role-systemd_service master: Add option to create systemd native service overrides https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/765815 | 10:29 |
| Adri2000 | hello! anyone knows if using the user_external_repos_list variable is deprecated? if it's been replaced by another mechanism? asking because https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/master/defaults/main.yml#L58-L60 mentions "the *old* override named user_external_repo_lists" | 10:33 |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 10:40 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance master: Add glance cache variables into glance-api https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/770606 | 10:41 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/769945 | 10:53 |
| *** gshippey has joined #openstack-ansible | 10:56 | |
| *** spatel has joined #openstack-ansible | 11:25 | |
| *** spatel has quit IRC | 11:29 | |
| jrosser | noonedeadpunk: this is what devstack has a patch for in order to install tempest from source https://review.opendev.org/c/openstack/devstack/+/765149/2/inc/python | 12:01 |
| jrosser | any thoughts? | 12:01 |
| noonedeadpunk | uh. that seems a bit tough.... | 12:25 |
| noonedeadpunk | I think in our case, u-c should be working as contraints? | 12:26 |
| noonedeadpunk | *constraints | 12:26 |
| noonedeadpunk | and the main problem with new resolver is that there should not be links, and you eliminated this with moving them to requirements in roles | 12:29 |
| noonedeadpunk | so it's kind of just needs to mention all packages that we don't want to get from pip but installed from soure? | 12:29 |
| jrosser | i'm not excatly sure what is happening | 12:30 |
| jrosser | but for certain when you have tempest==<version> at the u_c url you cannot install from git source at all | 12:30 |
| noonedeadpunk | and we want to install from source for master only | 12:31 |
| jrosser | i'm just looking at duplicating what they have done for devstack and download/adjust u_c before giving it to python_venv_build | 12:31 |
| jrosser | yes | 12:31 |
| jrosser | this is what happens in the wheel build http://paste.openstack.org/show/801622/ | 12:32 |
| jrosser | and it JustWorks(tm) as soon as i comment out the tempest constraint from u_c | 12:33 |
| *** akahat|lunch is now known as akahat|rover | 12:36 | |
| noonedeadpunk | well, for tempest we could ofc just decide to drop installing it from source and ease logic of the role dramatically. But that might influence other things, like clients | 12:36 |
| noonedeadpunk | (despite we don't install them from source right now) | 12:36 |
| *** rfolco has left #openstack-ansible | 12:36 | |
| noonedeadpunk | and we cant just drop all packages from global-constraints that are mentioned in requirements.txt | 12:39 |
| noonedeadpunk | so we need to parse only the ones that are defined as links and get package name and remove from global-constraints or smth like that, uh :( | 12:39 |
| jrosser | something like that, yes | 12:41 |
| jrosser | maybe right now this only affects tempest | 12:41 |
| jrosser | main issue really is that today we pass the u_c URL direct to pip, so this needs changing in the role | 12:41 |
| jrosser | well not quite actually | 12:42 |
| jrosser | the u_c URL gets put in a file, passed to pip with --constraint | 12:42 |
| jrosser | i'm just going to try to hack something up quick inside os_tempest then we should look at the code and decide if this is a thing we want to do in roles, or in python_venv_build | 12:43 |
| jrosser | fg | 12:44 |
| noonedeadpunk | that feel like being nasty thing in any scenario | 12:47 |
| noonedeadpunk | maybe we should really drop installing tempest from source especially considering we do this only for master | 12:50 |
| *** rh-jelabarre has joined #openstack-ansible | 13:02 | |
| noonedeadpunk | we would be exposed to the risk of getting broken tempest from pip then, but we can probably use global-requirement-pins.txt to temp fallback to previous version of tempest | 13:03 |
| *** spatel has joined #openstack-ansible | 13:13 | |
| *** spatel has quit IRC | 13:18 | |
| *** macz_ has joined #openstack-ansible | 13:19 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Remove neutron_keepalived_no_track variable https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/770808 | 13:22 |
| *** macz_ has quit IRC | 13:24 | |
| *** L_SR has joined #openstack-ansible | 13:42 | |
| L_SR | Hello Community! I'm experiencing a strange error while executing setup-hosts.yml playbook regarding the inability of infra1-* containers to ssh to infra1 node. This issue happens particularly at the lxc_containers_create/lxc_container_config.yml task. I can avoid it by setting lxc_container_journal_link : false, but then the issue resurfaces when | 13:47 |
| L_SR | running setup-infrastructure.yml. Any clues? | 13:47 |
| L_SR | My setup is composed of LXD VMs, with OSA deployment node in the same br-mgmt L2 network as the target nodes. This is not my first time using OSA, but definitely the first time experiencing this. Any ideas? | 13:47 |
| L_SR | Some quick notes: infra1-* containers DO resolve infra1 IP address correctly. Inventory seems to be fine. OUC is nothing fancy, in fact is the exact copy (with changed network config) of another VM-based deployment. Target nodes and Deployment nodes are LXC VMs physically in the same nodes. Networking among LXD VMs uses linuxbridges managed by LXD. | 13:49 |
| L_SR | Using TRAIN, btw | 13:50 |
| L_SR | OUC: https://pastebin.com/uVfB2yxd, Inventory: https://pastebin.com/h3nxMCww | 13:55 |
| L_SR | Thanks in advance for your help! | 13:59 |
| jrosser | could you paste the error you get, not sure when the infra containers should ssh to the hosts? | 14:00 |
| *** spatel has joined #openstack-ansible | 14:00 | |
| jrosser | the error and the surrounding context, so we know where we are in the plays | 14:00 |
| L_SR | jrosser thanks for your time. You may find the log from setup-hosts here: https://pastebin.com/J20CyXK0 | 14:09 |
| L_SR | ~line 1318 | 14:09 |
| jrosser | heres the task https://opendev.org/openstack/openstack-ansible-lxc_container_create/src/branch/master/tasks/lxc_container_config.yml#L272-L282 | 14:11 |
| jrosser | ultimately the error is "Could not resolve hostname infra1: Temporary failure in name resolution" | 14:12 |
| L_SR | yes | 14:12 |
| spatel | Its mandatory to have DNS name for all infra* nodes. | 14:14 |
| L_SR | OSA updates /etc/hosts accordingly. That is, any target not can indeed resolve infra1 | 14:15 |
| spatel | jrosser: I have found if you deploy HAproxy on separate box then some services won't work like Senlin. | 14:15 |
| L_SR | infra1-* containers are resolved by lxcbr0 bridge | 14:15 |
| jrosser | spatel: thats a bug :) | 14:15 |
| jrosser | spatel: i have seperate haproxy and everything works, so i'd guess its specific to senlin | 14:15 |
| spatel | i spent couple hours to figure that out :( | 14:15 |
| jrosser | i don't have senlin deployed though, so cant check that | 14:16 |
| spatel | I am going to talk to senlin make it work for internal URL instead of public :) | 14:16 |
| spatel | default it use public | 14:16 |
| jrosser | oh well thats a different issue | 14:16 |
| spatel | totally not blaming HAproxy :) | 14:17 |
| jrosser | "bug in OSA not wiring haproxy right" vs "senlin endpoints are broken in their code" | 14:17 |
| jrosser | sorry thought you meant the first | 14:17 |
| jrosser | i think we already knew they mess up endpoints as it doesnt work so well for the OSA tempest tests | 14:17 |
| spatel | we are good nothing wrong with OSA deployment. (I am going to work with senlin folks to see how we can make that change so it work with internalURL ) | 14:18 |
| jrosser | yes, please do | 14:18 |
| jrosser | are you doing this in irc with them? | 14:18 |
| spatel | First opening bug and then IRC | 14:19 |
| jrosser | L_SR: i'm not sure what is happening, you could run with -vvvv and it gives a lot of detail about how the connection plugin is deciding where to connect to | 14:19 |
| L_SR | jrosser thanks! Will run again with verbosity to see if I can spot something different | 14:20 |
| *** dave-mccowan has joined #openstack-ansible | 14:20 | |
| spatel | jrosser: where is setting or variable coming from in octavia? amp_flavor_id = a2102cf6-f7e4-427e-a42c-6187e12da548 | 14:24 |
| spatel | I want to change ID | 14:24 |
| spatel | its not part of main.yml - https://docs.openstack.org/openstack-ansible-os_octavia/latest/ | 14:24 |
| jrosser | the playbook finds it it's not defined https://github.com/openstack/openstack-ansible-os_octavia/blob/master/tasks/octavia_post_install.yml#L45-L71 | 14:26 |
| jrosser | i think the documentation for that is missing from defaults/main.yml ? | 14:27 |
| spatel | that variable isn't part of defaults/main.yml | 14:28 |
| jrosser | unless actually you are supposed to change this https://github.com/openstack/openstack-ansible-os_octavia/blob/master/defaults/main.yml#L262 | 14:28 |
| jrosser | and then it will look up the uuid for the new name | 14:28 |
| spatel | i didn't test that, i can give it a try to that var | 14:29 |
| spatel | one more bug, i set service_region: Phoenix in user_varaibles.yml and deployed all services | 14:30 |
| spatel | in openstack endpoint list i found senlin still using RegionOne | 14:31 |
| spatel | i tried to find issue but everything looks good in roles so not sure why its not accepting service_region: Phoenix | 14:31 |
| *** dwilde has joined #openstack-ansible | 14:32 | |
| *** pto has quit IRC | 14:33 | |
| spatel | I can see its here - https://github.com/openstack/openstack-ansible-os_senlin/blob/master/tasks/service_setup.yml#L154 | 14:34 |
| jrosser | there is a patch for service region https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/769956 | 14:35 |
| spatel | jrosser: you guys beat me :) | 14:36 |
| *** CeeMac has joined #openstack-ansible | 14:39 | |
| *** L_SR has quit IRC | 14:44 | |
| jrosser | noonedeadpunk: i got tempest to install / run, heres the change i needed http://paste.openstack.org/show/801623/ | 14:54 |
| jrosser | another horrid problem with stackviz too | 14:54 |
| jrosser | theres no python3.8 version of netifaces https://pypi.org/project/netifaces/#files | 14:55 |
| *** L_SR has joined #openstack-ansible | 14:56 | |
| jrosser | previously this is already build in the repo server when the tempest venv is made | 14:56 |
| jrosser | but for stackviz currently we set venv_wheel_build_enable: no | 14:56 |
| jrosser | which means it doesnt look at any of the already built wheels | 14:56 |
| jrosser | and that flag seems to be there specifically to address stackviz https://github.com/openstack/ansible-role-python_venv_build/commit/b84199341f1e13b43b8754dadafda219c16259d8 | 14:57 |
| *** L_SR has quit IRC | 15:02 | |
| jrosser | spatel: you've done sriov? | 15:05 |
| spatel | Yes a lot | 15:05 |
| *** dwilde has quit IRC | 15:06 | |
| spatel | 80% workload running on sriov :) | 15:06 |
| jrosser | did you have to adjust the neutron policy to allow non-admin users to make vnic_type=direct ports | 15:06 |
| *** dwilde has joined #openstack-ansible | 15:07 | |
| spatel | I don't think so i did anything | 15:07 |
| spatel | i believe i made some adjustment in policy file. i need to look | 15:07 |
| spatel | you are saying non-admin can't create neutron port direct. right? | 15:08 |
| spatel | I can give it a shot in few min.. i just build fresh cluster without any policy edit, let me create port and get back to you | 15:09 |
| jrosser | ok cool, it seems that way (non admin cant make port direct) just want to sanity check | 15:09 |
| spatel | looking at my policy file. hold on | 15:10 |
| spatel | jrosser: this is what my policy.json looks like - http://paste.openstack.org/show/801624/ | 15:13 |
| spatel | This is production one | 15:14 |
| jrosser | "create_port:binding:vnic_type": "rule:regular_user" <- ahha | 15:14 |
| spatel | "create_port:binding:vnic_type": "rule:regular_user", | 15:15 |
| spatel | gt it | 15:15 |
| spatel | got* | 15:15 |
| jrosser | awesome thankyou :) | 15:15 |
| spatel | does OSA allow to modify policy using variables? because in this case i did by hand :( | 15:16 |
| jrosser | https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/neutron_post_install.yml#L66-L71 | 15:18 |
| spatel | hmm interesting, what would be the overrides syntax for "create_port:binding:vnic_type": "rule:regular_user"? | 15:20 |
| spatel | do you have any example? | 15:20 |
| jrosser | not right now - if we fix our sriov thing via policy overrides then maybe later | 15:21 |
| jrosser | andrewbonney is taking a look at this now | 15:21 |
| spatel | thank you! | 15:23 |
| *** L_SR has joined #openstack-ansible | 15:26 | |
| L_SR | jrosser just found out that by adding the problematic domain to OSA deployment node's /etc/hosts resolved the issue. | 15:27 |
| L_SR | that is: added infra1 to /etc/hosts | 15:29 |
| jrosser | do you have a seperate deployment host? | 15:30 |
| L_SR | yes | 15:31 |
| L_SR | target hosts are LXD VM, while deployment node is just an LXC container in the same L2 network | 15:31 |
| jrosser | ok, OSA doesnt touch the /etc/hosts on the deploy hosts | 15:31 |
| spatel | L_SR: I have learnt that hard way, my all deployment has separate box and found you have to add DNS for all infra* | 15:31 |
| L_SR | spatel I guess I've earn my scar :). Been trying for a couple of days now. | 15:32 |
| spatel | jrosser: may be we should add some check and modify error messages :) | 15:32 |
| L_SR | thank you for helping me out | 15:32 |
| jrosser | could have something in setup-hosts to try to resolve the targets if the deploy host is not in the inventory | 15:33 |
| spatel | +1 | 15:33 |
| jrosser | it feels like a specific delegate_to issue though | 15:34 |
| jrosser | becasue before that point there will have been tasks run against infra* | 15:34 |
| spatel | why don't we add all hosts in deployment node /etc/hosts file | 15:34 |
| L_SR | I'm using Virtoria; the strange issue is that in my other deployment running Train, infra1 is not in deployment node's /etc/hosts | 15:35 |
| L_SR | I'm using Victoria; the strange issue is that in my other deployment running Train, infra1 is not in deployment node's /etc/hosts | 15:35 |
| spatel | I had no DNS issue till stein deployment, this is something new | 15:36 |
| *** macz_ has joined #openstack-ansible | 15:36 | |
| L_SR | @spatel same here | 15:36 |
| spatel | after stein i found DNS is required | 15:36 |
| jrosser | well its a different major version of ansible for a start | 15:36 |
| jrosser | that could change all sorts | 15:36 |
| spatel | jrosser: why don't we take inventory file and push into /etc/hosts file of deployment node? | 15:38 |
| jrosser | theres very much less churn now around 2.9 / 2.10 but for rocky/stein kind of times upgrading from ansible 2.N to 2.N+1 was a really huge work | 15:38 |
| *** Muran has quit IRC | 15:39 | |
| spatel | This is not a very big issue but sometime hard to debug based on error message :) If person isn't in this channel then it would be fun to debug. | 15:40 |
| spatel | Anyway we can make it clear this clear in official document so folks will be aware | 15:40 |
| spatel | jrosser: ready for this? | 15:42 |
| spatel | https://review.opendev.org/c/openstack/senlin/+/749874 | 15:42 |
| spatel | patch is there for senlin | 15:42 |
| jrosser | oh awesome | 15:43 |
| spatel | look like we need to tell osa to obtain that | 15:43 |
| jrosser | you sure, it merged in october? | 15:43 |
| *** L_SR has quit IRC | 15:45 | |
| spatel | https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/defaults/repo_packages/openstack_services.yml#L188 | 15:47 |
| jrosser | spatel: the issue is that the patch merged into senlin *after* the openstack victoria release was made | 15:47 |
| spatel | Yes! | 15:48 |
| jrosser | so today that patch is only on senlin master branch | 15:48 |
| spatel | that is correct | 15:48 |
| jrosser | you will need to override the version of senlin you are deploying | 15:48 |
| spatel | That is what i am going to do :) | 15:48 |
| spatel | may be in RC2 release we can bump that in OSA | 15:48 |
| jrosser | well no we can't because OSA victoria will deploy the victoria version of openstack | 15:49 |
| spatel | I meant in next release we can include that patch using senlin_git_install_branch: right? | 15:51 |
| jrosser | that patch does not exist on the stable/victoria branch of the senlin project | 15:55 |
| spatel | That is what i am trying to find. | 15:56 |
| spatel | what is the commit it in master? | 15:56 |
| spatel | i want to override senlin_git_install_branch: | 15:57 |
| spatel | got it - https://opendev.org/openstack/senlin/commit/d4ec93ae55256e8ccf04eac45c5364c694a34759 | 15:59 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Move tempest pip package from a constraint to a requirement https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/770281 | 16:00 |
| *** cloudnull has joined #openstack-ansible | 16:05 | |
| spatel | jrosser: -e venv_rebuild=yes this correct option to re-build senlin right? | 16:14 |
| *** miloa has quit IRC | 16:17 | |
| *** gyee has joined #openstack-ansible | 16:22 | |
| admin0 | how does openstack instance gets dual ip stack ? if both ipv4 and v6 are added ? | 16:24 |
| spatel | Can you take a look what is wrong here - http://paste.openstack.org/show/801629/ | 16:24 |
| admin0 | do we add both in the same net ? | 16:24 |
| spatel | Yes on same nat admin0 | 16:24 |
| spatel | i am running dual stack | 16:24 |
| admin0 | i added ipv4 as ext-net on vlan101 .. how do I add v6 there ? does v6 also need to be added on vlan 101 ? | 16:25 |
| spatel | jrosser: i have rebuild senlin but still i am not seeing desired changed | 16:25 |
| spatel | just add ipv6 subnet in existing ipv4 vlan101 network. | 16:25 |
| spatel | v6 doesn't need dedicated network if you doing dual stack | 16:26 |
| spatel | Just | 16:28 |
| spatel | neutron subnet-create net_pub_vlan_100 2601:c05:3011::1/64 --name sub_pub6_vlan_100 --allocation-pool start=2601:c05:3011::10,end=2601:0c05:3011:0000:ffff:ffff:ffff:ffff --gateway=2601:c05:3011::1 --description "Public IPv6 Subnet" --ip-version=6 | 16:28 |
| spatel | jrosser: noonedeadpunk any idea why senlin failing here - http://paste.openstack.org/show/801630/ | 16:43 |
| spatel | full error logs - http://paste.openstack.org/show/801631/ | 16:44 |
| noonedeadpunk | jrosser: `venv_build_constraints: "{{ (tempest_git_repo is defined) | ternary( _u_c_contents.content.split('\n') | reject('match', '[tempest=*]') | list, tempest_git_constraints) }}"` looks not as horrific as I expected | 17:08 |
| jrosser | yes, could have been worse i think | 17:09 |
| noonedeadpunk | only a bit of afraid that we keep all u_c in memory | 17:09 |
| jrosser | it's not really so bad, maybe 50 lines | 17:09 |
| jrosser | but i am not knowing what to do about stackviz | 17:10 |
| jrosser | i think the patch will still fail | 17:10 |
| noonedeadpunk | you wanted to say 700? | 17:10 |
| jrosser | hmm | 17:11 |
| jrosser | maybe i wanted to say 349 | 17:11 |
| jrosser | cat tempest-22.0.0.0rc2.dev13-source-constraints.txt | wc -l | 17:12 |
| jrosser | 349 | 17:12 |
| noonedeadpunk | hm... https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt is 569 lines long.... | 17:12 |
| noonedeadpunk | just train was 700 | 17:12 |
| jrosser | look at line 22-28 here http://paste.openstack.org/show/801623/ | 17:13 |
| jrosser | this is also horrid | 17:13 |
| noonedeadpunk | yeah. I guess it's regarding stakwiz, rigt? | 17:13 |
| jrosser | indeed | 17:13 |
| noonedeadpunk | I have no idea here tbh atm... | 17:13 |
| jrosser | i didnt commit that yet because, well, its gross | 17:13 |
| jrosser | and i am not sure it is even going to work for the tripleo people becasue they don't have the repo server anyway | 17:14 |
| noonedeadpunk | indeed | 17:14 |
| jrosser | it fails on wanting python.h | 17:14 |
| jrosser | so one answer is to install all the necessary -dev packages on the utility host | 17:14 |
| jrosser | which we only just finished cleaning out :/ | 17:15 |
| jrosser | becasue there is no suitable package for netifaces in pypi just just goes ahead and builds it from source | 17:15 |
| noonedeadpunk | well, I think then os_tempest should go on and install these dev packages? | 17:16 |
| noonedeadpunk | but yeah... | 17:16 |
| noonedeadpunk | I see what you mean | 17:16 |
| noonedeadpunk | Is stackwiz owned by tripleo? | 17:17 |
| jrosser | could be - seems kind of wierd install from tarball | 17:18 |
| jrosser | its not on pypi | 17:19 |
| spatel | jrosser: noonedeadpunk any idea what is going on - http://paste.openstack.org/show/801631/ | 17:21 |
| spatel | i tried senlin_git_install_branch: d4ec93ae55256e8ccf04eac45c5364c694a34759 | 17:21 |
| spatel | look like some dependency issue for that patch | 17:22 |
| jrosser | my first guess is that the build is not happening on the repo server | 17:22 |
| jrosser | where is that log from? | 17:22 |
| noonedeadpunk | oh, yes, it's not running on repo server | 17:23 |
| noonedeadpunk | so container is missing pkg-config | 17:23 |
| jrosser | spatel: you have the repo server? | 17:24 |
| spatel | yes i do have repo | 17:24 |
| spatel | i can install apt-get install pkg-config and see | 17:24 |
| jrosser | well ok, so the error is "why does it not build there" rather than anything to do with senlin | 17:24 |
| jrosser | nooooooo | 17:24 |
| jrosser | fix the cause :) | 17:24 |
| spatel | i am not following you jrosser | 17:25 |
| jrosser | noonedeadpunk and i are both saying "this does not look like the build is happening on the repo server" | 17:25 |
| jrosser | and it should be doing | 17:25 |
| jrosser | thats why i ask where exactly do you get that log from? | 17:26 |
| spatel | i am using this command openstack-ansible os-senlin-install.yml --limit ostack-phx-api-1-1_senlin_container-314663f9 | 17:26 |
| spatel | do you think that coule be the issue because of --limit? | 17:26 |
| jrosser | this maybe the curse of --limit again | 17:26 |
| spatel | let me remove limit and try hold on | 17:26 |
| spatel | i was tying to test on just infra-1 so i did that but let me go with all | 17:27 |
| noonedeadpunk | you should be limiting infra-1 and repo_all or smth then | 17:30 |
| spatel | look like it moved further so limit was culprit | 17:30 |
| spatel | damn it! | 17:31 |
| spatel | Thank you! | 17:31 |
| noonedeadpunk | btw, I guess we can now try out tempest for senlin again | 17:31 |
| spatel | indeed | 17:33 |
| spatel | let me try with openstack-ansible os-senlin-install.yml -e venv_rebuild=yes | 17:33 |
| spatel | i am not seeing required changes so look like it didn't use patch branch | 17:34 |
| spatel | noonedeadpunk: it didn't pull that repo branch which i specified in user_variables.yml | 17:40 |
| spatel | senlin_git_install_branch: d4ec93ae55256e8ccf04eac45c5364c694a34759 | 17:40 |
| spatel | it has rebuild it but i am looking at file conf/authentication.py it doesn't have any changes required to specify interface | 17:41 |
| spatel | anyway to verify which branch it used to pull source code? | 17:42 |
| *** rpittau is now known as rpittau|afk | 17:42 | |
| spatel | let me destroy container and try | 17:46 |
| *** pcaruana has quit IRC | 17:51 | |
| spatel | bummer!! noonedeadpunk i have destroyed all container and re-run playbook with limit and hit same error :( | 17:54 |
| spatel | sorry without* limit | 17:55 |
| spatel | i have tried to install apt-get install pkg-config but still same error | 17:57 |
| spatel | I don't have this file /openstack/venvs/senlin-22.0.0.0rc1/global-constraints.txt | 18:00 |
| spatel | look like some previous task missed something | 18:00 |
| *** pcaruana has joined #openstack-ansible | 18:03 | |
| *** andrewbonney has quit IRC | 18:17 | |
| *** maharg101 has quit IRC | 18:17 | |
| spatel | where does osa checkout git source code? | 18:20 |
| spatel | i want to check why its not checking out master commit? | 18:20 |
| jrosser | everything happens in the python_venv_build role | 18:21 |
| jrosser | follow the trail of where senlin_git_constraints goes | 18:21 |
| spatel | that means on senlin container itself | 18:21 |
| jrosser | no | 18:21 |
| jrosser | with a repo server the build of the python wheels from source code happens on the repo server, by delegation | 18:22 |
| jrosser | pay careful attention to the delegation [this -> that} format in the ansible output | 18:22 |
| jrosser | it would really be worthwhile understanding the whole of whats happening here, as it's fundamental | 18:23 |
| jrosser | there is an excellent readme on this https://github.com/openstack/ansible-role-python_venv_build | 18:23 |
| spatel | agreed i need to understand basic flow. | 18:24 |
| spatel | reading your link now. | 18:26 |
| spatel | I am trying to find why senlin not checkout specified branch | 18:26 |
| jrosser | first thing that happens is wheels get built | 18:36 |
| jrosser | so move the wheel build log aside on the repo server (just to make things more obvious) | 18:36 |
| jrosser | run the play again, and you should be able to find the git versions make their way though to that log | 18:36 |
| spatel | sorry i wasn't looking at chat | 18:46 |
| spatel | jrosser: let me check where wheel logs | 18:47 |
| spatel | jrosser: this one right ? /var/log/python_venv_build.log on repo | 18:48 |
| spatel | let me do that and re-run | 18:49 |
| spatel | jrosser: i have moved wheel on repo and re-run playbook | 18:57 |
| spatel | it does re-create log file and i am not seeing any git branch version in logs | 18:58 |
| spatel | even not which comes with OSA | 18:58 |
| spatel | i meant - https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/defaults/repo_packages/openstack_services.yml#L188 | 18:58 |
| spatel | is it possible its not downloading new one and just using cached one | 18:59 |
| jrosser | spatel: i don't know, the wheel build log will tell you whats happening | 19:01 |
| jrosser | follow the vars & code | 19:01 |
| spatel | doing experiment, i have remove senlin repo location from default/openstack_services.yml | 19:05 |
| spatel | i want to see if it fail or not | 19:06 |
| spatel | I am not seeing anywhere in logs or playbook run out put where it downloading fresh git checkout | 19:06 |
| spatel | that is interesting, i have removed senlin_git_repo from this file - playbooks/defaults/repo_packages/openstack_services.yml | 19:12 |
| spatel | but still i am able to deploy senlin without error | 19:13 |
| spatel | let me destroy all contain and see where that thing is cached | 19:14 |
| spatel | i gave up :( | 19:29 |
| spatel | i am going to move my HAproxy to F5 loadbalancer for now | 19:30 |
| spatel | will talk to noonedeadpunk if i missed something | 19:30 |
| jrosser | spatel: in the wheel build log like this https://zuul.opendev.org/t/openstack/build/4bffa026b29a420d9547ec647e2c79aa/log/logs/openstack/aio1_repo_container-9b3f1f5e/python_wheel_build.log.txt#18640 | 19:47 |
| jrosser | that is keystone for example | 19:47 |
| jrosser | you never see the git repo cloned specifically as the URL is just given to pip and it does all the wheel build stuff itself | 19:47 |
| spatel | jrosser: i can see all other stuff getting install | 19:48 |
| spatel | jrosser: where it will checkout full senlin branch? | 19:49 |
| jrosser | like i say you give the url with the SHA embedded to pip | 19:49 |
| jrosser | it downloads / builds / cleans up | 19:49 |
| jrosser | you never see the repo | 19:49 |
| spatel | it will use this SHA right ? senlin_git_install_branch: a2d9092c8bb0bb1635e3ebbece343e8eded5fec1 | 19:51 |
| spatel | I did same thing or neutron_git_install_branch: and it works for neutron. but not for senlin | 19:53 |
| spatel | neutron successfully deployed with specific override commit | 19:54 |
| spatel | It feel like we have bug in senlin somewhere, its not taking that override | 19:54 |
| jrosser | well thats what i mean about starting with that variable and debugging it though the code | 19:55 |
| jrosser | needs time spending on it to understand whats happening | 19:55 |
| spatel | yes, trying out on lab.. | 19:56 |
| odyssey4me | jrosser how's the conversion to the new pip constraints mechanism going? | 20:03 |
| odyssey4me | spatel IIRC, the venv should have a constraints file inside it | 20:04 |
| jrosser | oh now its funny you mention that | 20:04 |
| jrosser | i spent all day on it | 20:04 |
| jrosser | would like to have an opinion on it if you have any time | 20:04 |
| spatel | odyssey4me: it does has constrains file inside /openstack/venvs/senlin-22.0.0.0rc1 | 20:05 |
| odyssey4me | spatel you should see three .txt files inside the venv - https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_install.yml#L60-L98 | 20:05 |
| odyssey4me | jrosser sure, I'll try - I'm quite rusty :) | 20:06 |
| odyssey4me | heh, I thought of pouring vinegar on myself... then realised it sounded like a Def Leppard song, then felt a bit old | 20:06 |
| *** avagi has quit IRC | 20:06 | |
| jrosser | well this was the most obvious and smallest change to make https://review.opendev.org/q/topic:%22osa-new-pip%22+(status:open%20OR%20status:merged) | 20:07 |
| spatel | odyssey4me: sorry i may be exhausted and not following you. what do you means it should have constraints inside? | 20:07 |
| odyssey4me | spatel if you look inside the venv folder - are there three txt files? | 20:08 |
| spatel | if i destroy whole container and re-build fresh then it has nothing to do right? | 20:08 |
| spatel | on senlin container (yes they are there) | 20:08 |
| odyssey4me | ok, do the content have the repo and sha you're pinning to? | 20:08 |
| spatel | jrosser: odyssey4me i think i found the bug | 20:09 |
| odyssey4me | jrosser I remember trying to do that initially because I wanted only a requirements and global/upper constraints file... but there was a reason I didn't which may now be defunct | 20:10 |
| spatel | Ready for this? | 20:10 |
| spatel | https://opendev.org/openstack/openstack-ansible-os_senlin/src/branch/master/defaults/main.yml#L44 | 20:10 |
| spatel | #egg=openstack-senlin <-- that was the issue | 20:11 |
| spatel | i did test in lab but let me try this on other environment to verify | 20:11 |
| *** maharg101 has joined #openstack-ansible | 20:14 | |
| *** pcaruana has quit IRC | 20:15 | |
| jrosser | odyssey4me: well we kind of have no choice with the new resolver to move the source install url to the requirements | 20:15 |
| odyssey4me | jrosser if it works, then great - I think you *might* have to ensure that the UC has that egg name removed though, otherwise you have a conflict | 20:16 |
| jrosser | look at the patch against os_tempest ..... | 20:16 |
| odyssey4me | spatel yeah, that makes sense - the senlin package seems to be named -senlin' - not 'openstack-senlin' | 20:17 |
| jrosser | but there is a problem | 20:17 |
| jrosser | that we have two concepts of constraints | 20:17 |
| jrosser | during wheel build with my patch it uses u-c for constraints, which is good | 20:18 |
| *** maharg101 has quit IRC | 20:19 | |
| odyssey4me | ok, but does a constraint for, say, tempest in u-c override the requirement with git? | 20:19 |
| spatel | odyssey4me: yes now everything working :) | 20:19 |
| spatel | I am going to submit patch | 20:19 |
| jrosser | odyssey4me: worse, it just blows up saying "you cant do that" | 20:19 |
| jrosser | anyway, i've worked around that by excluding the u-c tempest, becasue thats a one off corner case | 20:19 |
| jrosser | problem i've now got is this https://github.com/openstack/ansible-role-python_venv_build/blob/master/tasks/python_venv_wheel_build.yml#L162-L169 | 20:20 |
| odyssey4me | jrosser right... so unfortunately that means you have to pull the u-c down, remove anything with the same name in any other constraint | 20:20 |
| jrosser | we generate/slurp a *new* set of constraints for venv install time | 20:20 |
| jrosser | which include 'horizon-x.y.xdevN" for example | 20:21 |
| jrosser | and also at the same time we have still the horizon source requirements URL in the requrements.txt | 20:21 |
| odyssey4me | jrosser right, that's for when there's a repo available | 20:21 |
| jrosser | and those are contradictory, though they are the same thing | 20:21 |
| odyssey4me | yep | 20:21 |
| jrosser | so again pip says "cant do that" | 20:21 |
| jrosser | so it's sort of the same problem that i had with tempest being in u-c | 20:22 |
| odyssey4me | so effectively the u-c needs to be pulled down and cleaned of any packages intended to be built from git | 20:22 |
| odyssey4me | then the wheels built | 20:22 |
| jrosser | but now the slurped constraints generated from the wheel build contain the thing which comes from the source install | 20:22 |
| odyssey4me | then the new constraints file compiled and the requirements then need to be cleaned | 20:22 |
| odyssey4me | it really is quite messy | 20:23 |
| jrosser | i guess we need to strip the git+... from the requirements | 20:23 |
| jrosser | for the venv install only | 20:23 |
| jrosser | when there is a repo sever | 20:23 |
| jrosser | .... :) complicated | 20:23 |
| odyssey4me | the requirements will have to be just the egg-name | 20:23 |
| jrosser | for venv build yes, | 20:23 |
| jrosser | but i think they need to stay as the git url for wheel build | 20:24 |
| odyssey4me | yep, if there's no repo then of course everything stays as-is, except the u-c needs cleaning up of the egg name | 20:24 |
| jrosser | so far i've only found tempest to be doing that | 20:24 |
| jrosser | stuff like glance/neutron doesnt appear there | 20:24 |
| jrosser | may be more hiding though | 20:24 |
| odyssey4me | well, yeah - the issue is that tempest is in u-c | 20:26 |
| odyssey4me | there are others that may lurk there - it changes from time to time | 20:26 |
| odyssey4me | so never mind the local dance of stuff generated by OSA - u-c itself needs cleansing too | 20:26 |
| odyssey4me | luckily there's a local copy | 20:27 |
| jrosser | i have a hack in here https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/770281/5/tasks/tempest_install_source.yml | 20:27 |
| odyssey4me | for a similar issue, I did this https://github.com/redhat-openstack/infrared/blob/master/infrared/common/roles/create_venv/tasks/main.yml#L119-L124 | 20:28 |
| odyssey4me | something like that could be put into the venv build role as a parameter | 20:29 |
| odyssey4me | then we just do this: https://github.com/redhat-openstack/infrared/blob/aa10309b94de66de7349c42f6047a965ba4310ac/plugins/tempest/tasks/setup/pip/main.yml#L16-L25 | 20:30 |
| odyssey4me | the general idea there is to keep the complexity on the role and keep the service role simpler | 20:34 |
| odyssey4me | tempest is horrible though, with all those plugins and such | 20:34 |
| * jrosser looks | 20:39 | |
| odyssey4me | ok folks, I'm out for the night - have a good one, chat tomorrow! | 20:56 |
| jrosser | o/ thanks for the pointers | 20:57 |
| spatel | odyssey4me: good night | 21:02 |
| spatel | jrosser: that patch didn't help :) problem is senlin core driver trying to reach out to other services using public URL | 21:03 |
| jrosser | as far as i know it can now take more parameters from the config file | 21:03 |
| spatel | current patch is just for keystone | 21:03 |
| jrosser | have you added settings for the endpoint? | 21:04 |
| spatel | I just added interface = internal and verify_ssl = False in senlin.conf | 21:04 |
| spatel | senlin directly talk to nova/neutron using whatever endpoint keystone handover to senlin | 21:05 |
| spatel | Let me do more testing to verify more. | 21:06 |
| *** jrosser has quit IRC | 21:29 | |
| *** poopcat1 has quit IRC | 21:30 | |
| *** poopcat1 has joined #openstack-ansible | 21:30 | |
| *** hindret has quit IRC | 21:30 | |
| *** hindret has joined #openstack-ansible | 21:30 | |
| *** jrosser has joined #openstack-ansible | 21:32 | |
| *** dwilde has quit IRC | 21:49 | |
| *** dwilde has joined #openstack-ansible | 21:51 | |
| *** poopcat1 is now known as poopcat | 22:20 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Bump Victoria for the release https://review.opendev.org/c/openstack/openstack-ansible/+/770073 | 22:22 |
| *** spatel has quit IRC | 22:27 | |
| *** gshippey has quit IRC | 23:48 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 23:49 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!