| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO https://review.openstack.org/567224 | 00:00 |
|---|---|---|
| *** noslzzp has quit IRC | 00:01 | |
| jtcressy | mwhahaha: BTW: If I try to update the stack to retry that step2 execution I got stuck on above, it cant handle a case where the containers are already existing. I have to fully re-deploy afterall. https://hastebin.com/lahuviquyu | 00:02 |
| mwhahaha | jtcressy: yea I guess you have to clear all the ceph nodes | 00:06 |
| mwhahaha | Sounds like a bug with ceph-ansible though | 00:07 |
| jtcressy | I think in the distant future, I may try to use something like croit.io to stand up the ceph cluster and just point cinder at it. I Imagine it'd be easier to manage than cli-only ceph-ansible. | 00:08 |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 00:10 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 00:10 |
| *** itlinux has quit IRC | 00:13 | |
| mwhahaha | pabelanger: so the issue with using the proxy for that file is if it gets stale and the repo isn't available it can have problems. Though we could try the proxy and see what happens. But we'll likely have other dlrn issues if rdoproject is unhappy | 00:21 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Take featureset out of TOCI_JOBTYPE https://review.openstack.org/582384 | 00:21 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Expose Horizon "DocumentRoot" on host https://review.openstack.org/566277 | 00:21 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add release note for vnx and unity template changes https://review.openstack.org/583094 | 00:21 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Increase default RabbitMQ/Erlang TCP timeout from 5 to 15 seconds https://review.openstack.org/576245 | 00:21 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Do undercloud container prepare in external_deploy_tasks https://review.openstack.org/581917 | 00:22 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Break out image prepare into its own "service" https://review.openstack.org/581918 | 00:22 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: WIP Always enable prepare service for docker clouds https://review.openstack.org/581919 | 00:22 |
| *** lblanchard has joined #tripleo | 00:23 | |
| pabelanger | mwhahaha: right, if the reverse proxy doesn't have the repo, then rdoproject upstream is down. If we've cached it, we won't expire it until 24 hours I think | 00:23 |
| pabelanger | will offer some protection | 00:23 |
| pabelanger | 24 hours from last access | 00:23 |
| pabelanger | we've had trunk.rdoproject.org got down for a few days, and not break tripleo upstream, because we've cached the files in apache | 00:24 |
| mwhahaha | pabelanger: but does it do a freshness check? Cause we don't want it to use a stale one all the time | 00:24 |
| gfidente | jtcressy that is an interesting bug to file against ceph-ansible | 00:24 |
| pabelanger | yes, it will check upstream each time, if upstream is online | 00:24 |
| gfidente | jtcressy we purposely moved to ceph-ansible because it is maintained by ceph community | 00:25 |
| gfidente | (from puppet-ceph) | 00:25 |
| pabelanger | if down, will serve the file in cache | 00:25 |
| jtcressy | I mean, it's no big deal. I imagine something like this would happen when it is a bit more complex than a single mistral workflow. However, one should've been able to retry the playbook and catch existing containers and re-create them or something. | 00:26 |
| gfidente | what is croit.io using behind the scenes for the deployment of the ceph cluster? | 00:26 |
| jtcressy | I'm not sure yet. I've barely scratched the surface with them. Another one i know of from Mirantis is decapod, and it uses Ansible behind the scenes. | 00:27 |
| gfidente | so it's probably ceph-ansible again | 00:28 |
| *** karthiks has joined #tripleo | 00:29 | |
| jtcressy | Yeah. My reason for using something like that would be to separate my ceph deployment workflow from the openstack deployment so that I dont have to keep deploy/destroy/deploy/destroy repeatedly until I iron out every little error in every workflow. Get the ceph deployment done so you can focus on the primary cluster. | 00:29 |
| jtcressy | I've spent this entire summer doing that very thing, and it's getting old. 3 months of this and I haven't even _once_ been successful and got an actual functioning cluster. | 00:30 |
| mwhahaha | jtcressy: yea that make sense and we do support an external ceph cluster | 00:30 |
| mwhahaha | jtcressy: so of you want I can try and help you out tomorrow I think your close | 00:31 |
| jtcressy | I'm attempting my 3rd deployment this afternoon right now. It's provisioning the nodes right now. in another hour or two i'll get to see if step2_execution fails again or by some miracle actually finishes the ceph deployment and moves on. | 00:32 |
| *** threestrands has quit IRC | 00:38 | |
| *** gfidente has quit IRC | 00:38 | |
| *** owalsh_ is now known as owalsh | 00:44 | |
| *** khyr0n has quit IRC | 00:49 | |
| *** rcernin_ has joined #tripleo | 00:56 | |
| *** bdodd has quit IRC | 00:56 | |
| *** rcernin has quit IRC | 00:58 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 01:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 01:10 |
| *** slaweq has joined #tripleo | 01:11 | |
| *** dmacpher has quit IRC | 01:12 | |
| *** mrsoul` has joined #tripleo | 01:15 | |
| *** slaweq has quit IRC | 01:15 | |
| *** links has joined #tripleo | 01:17 | |
| *** mrsoul has quit IRC | 01:18 | |
| *** etingof has quit IRC | 01:22 | |
| *** Petersingh has joined #tripleo | 01:22 | |
| jtcressy | mwhahaha: Are the ceph monitors launched in containers or systemd units? The only error i'm getting now is it times out when trying to talk to the ceph monitors, but I dont see the monitor as a container on one of the cephstorage nodes. Only containers named "ceph-osd-prepare-overcloud-cephstorage-0-sd[b-g]" | 01:26 |
| jtcressy | one container per OSD | 01:26 |
| *** edmondsw has joined #tripleo | 01:27 | |
| jtcressy | checked the other two nodes and it's the same deal. | 01:28 |
| mwhahaha | jtcressy: containers | 01:28 |
| mwhahaha | jtcressy: are they on the controllers? | 01:28 |
| jtcressy | let me check... | 01:29 |
| jtcressy | here's the tail end of the mistral ceph log, grepped for "fail": https://hastebin.com/pewosarofa | 01:30 |
| *** yamahata has quit IRC | 01:31 | |
| jtcressy | ah, the monitors are on the controllers. | 01:31 |
| jtcressy | the containers are running however... | 01:31 |
| *** edmondsw has quit IRC | 01:31 | |
| jtcressy | The log looks like it's healthy. I'm only looking at the monitor on controller-0 so far | 01:32 |
| *** stendulker has joined #tripleo | 01:32 | |
| jtcressy | Very last line of the log on one of the containers shows this: "mon.overcloud-controller-0@1(peon).data_health(10) update_stats avail 96% total 272 GB, used 9721 MB, avail 262 GB" | 01:33 |
| jtcressy | So it's recognizing storage space. | 01:33 |
| mwhahaha | Bad net config maybe? | 01:33 |
| mwhahaha | Not sure | 01:33 |
| jtcressy | hmm, possible, but that would mean the monitors would not see any OSDs / not recognize storage space? | 01:33 |
| openstackgerrit | Merged openstack/tripleo-validations master: Adds heat-manage purge_deleted cron job validation https://review.openstack.org/577150 | 01:34 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add some air - blank line between network's in net configs https://review.openstack.org/580224 | 01:34 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: ControlPlaneSubnetCidr using get_attr https://review.openstack.org/579579 | 01:34 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: ControlPlaneDefaultRoute using get_attr https://review.openstack.org/579580 | 01:34 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Enable logging to stdout/stderr in memcached https://review.openstack.org/583344 | 01:34 |
| *** etingof has joined #tripleo | 01:35 | |
| jtcressy | the output of that mistral log is so ridiculously long. It'd definitely help to have some sort of summary entry in between those. I'm still thoroughly confused as to why it timed out accessing the monitor. | 01:39 |
| jtcressy | mwhahaha: Oh this is very concerning: https://hastebin.com/abawahavay | 01:41 |
| jtcressy | notice the lack of IP addresses | 01:41 |
| jtcressy | There should be a bridge on enp7s0d1 | 01:42 |
| jtcressy | i'm going to verify my network configs. | 01:42 |
| *** medberry has joined #tripleo | 01:43 | |
| *** medberry has quit IRC | 01:43 | |
| *** medberry has joined #tripleo | 01:43 | |
| jtcressy | :::major facepalm::: no configuration for the ceph-storage role in the network environment files. | 01:43 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-docs master: Document "openstack tripleo container image prepare" https://review.openstack.org/553104 | 01:47 |
| *** Petersingh is now known as Petersingh|afk | 01:48 | |
| jtcressy | This project has been an absolute lifesaver for setting up network isolation: https://github.com/cybertron/tripleo-scripts | 01:49 |
| *** jtcressy has quit IRC | 01:52 | |
| *** colonwq has quit IRC | 01:59 | |
| *** medberry has quit IRC | 02:00 | |
| *** stendulker has quit IRC | 02:04 | |
| *** lblanchard has quit IRC | 02:06 | |
| *** Petersingh|afk is now known as Petersingh | 02:06 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 02:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 02:10 |
| *** slaweq has joined #tripleo | 02:11 | |
| *** colonwq has joined #tripleo | 02:12 | |
| *** slaweq has quit IRC | 02:16 | |
| openstackgerrit | Ian Main proposed openstack/tripleo-common master: WIP: Use process-templates.py and cache templates to speed up plan ops. https://review.openstack.org/581153 | 02:31 |
| openstackgerrit | Ian Main proposed openstack/python-tripleoclient master: WIP: Speed up plan operations by using a tarball in swift. https://review.openstack.org/581141 | 02:31 |
| openstackgerrit | Dan Sneddon proposed openstack/os-net-config master: Stub out check for OVS installed to avoid failing tests https://review.openstack.org/585070 | 02:37 |
| *** eck` is now known as eck`gone | 02:47 | |
| openstackgerrit | Dan Sneddon proposed openstack/os-net-config master: Stub out check for OVS installed to avoid failing tests https://review.openstack.org/585070 | 02:50 |
| *** psachin` has joined #tripleo | 02:50 | |
| openstackgerrit | Dan Sneddon proposed openstack/os-net-config master: Stub out check for OVS installed to avoid failing tests https://review.openstack.org/585070 | 02:52 |
| *** brault has quit IRC | 02:53 | |
| *** yamahata has joined #tripleo | 02:57 | |
| *** brault has joined #tripleo | 02:57 | |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-aodh master: WIP: Add required role repo contents https://review.openstack.org/585075 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-barbican master: WIP: Add required role repo contents https://review.openstack.org/585076 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-ceilometer master: WIP: Add required role repo contents https://review.openstack.org/585077 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-cinder master: WIP: Add required role repo contents https://review.openstack.org/585078 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-congress master: WIP: Add required role repo contents https://review.openstack.org/585079 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-designate master: WIP: Add required role repo contents https://review.openstack.org/585080 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-glance master: WIP: Add required role repo contents https://review.openstack.org/585081 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-gnocchi master: WIP: Add required role repo contents https://review.openstack.org/585082 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-haproxy master: WIP: Add required role repo contents https://review.openstack.org/585083 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-heat master: WIP: Add required role repo contents https://review.openstack.org/585084 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-horizon master: WIP: Add required role repo contents https://review.openstack.org/585085 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-ironic master: WIP: Add required role repo contents https://review.openstack.org/585086 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-keepalived master: WIP: Add required role repo contents https://review.openstack.org/585087 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-manila master: WIP: Add required role repo contents https://review.openstack.org/585088 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-memcached master: WIP: Add required role repo contents https://review.openstack.org/585089 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-mistral master: WIP: Add required role repo contents https://review.openstack.org/585090 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-neutron master: WIP: Add required role repo contents https://review.openstack.org/585091 | 02:58 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-nova master: WIP: Add required role repo contents https://review.openstack.org/585092 | 02:59 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-octavia master: WIP: Add required role repo contents https://review.openstack.org/585093 | 02:59 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-opendaylight master: WIP: Add required role repo contents https://review.openstack.org/585094 | 02:59 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-ovn master: WIP: Add required role repo contents https://review.openstack.org/585095 | 02:59 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-panko master: WIP: Add required role repo contents https://review.openstack.org/585096 | 02:59 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-qdrouterd master: WIP: Add required role repo contents https://review.openstack.org/585097 | 02:59 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-rabbitmq master: WIP: Add required role repo contents https://review.openstack.org/585098 | 02:59 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-rsyslog-sidecar master: WIP: Add required role repo contents https://review.openstack.org/585099 | 02:59 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-sahara master: WIP: Add required role repo contents https://review.openstack.org/585100 | 02:59 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-sensu master: WIP: Add required role repo contents https://review.openstack.org/585101 | 02:59 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-swift master: WIP: Add required role repo contents https://review.openstack.org/585102 | 02:59 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-tacker master: WIP: Add required role repo contents https://review.openstack.org/585103 | 02:59 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-tempest master: WIP: Add required role repo contents https://review.openstack.org/585104 | 02:59 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-ui master: WIP: Add required role repo contents https://review.openstack.org/585105 | 02:59 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-zaqar master: WIP: Add required role repo contents https://review.openstack.org/585106 | 02:59 |
| openstackgerrit | Yong Huang proposed openstack/tripleo-heat-templates stable/queens: Add release note for vnx and unity template changes https://review.openstack.org/585107 | 03:01 |
| *** ramishra has joined #tripleo | 03:01 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 03:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 03:10 |
| *** jtcressy has joined #tripleo | 03:14 | |
| *** edmondsw has joined #tripleo | 03:15 | |
| *** edmondsw has quit IRC | 03:19 | |
| *** dmacpher has joined #tripleo | 03:23 | |
| *** dmacpher has quit IRC | 03:24 | |
| *** dmacpher has joined #tripleo | 03:26 | |
| *** psahoo has joined #tripleo | 03:28 | |
| *** Petersingh is now known as Petersingh|afk | 03:31 | |
| *** adrianreza has quit IRC | 03:34 | |
| *** holser_ has joined #tripleo | 03:35 | |
| *** bdodd has joined #tripleo | 03:44 | |
| *** bdodd has quit IRC | 03:48 | |
| *** bdodd has joined #tripleo | 03:52 | |
| *** saneax has quit IRC | 03:53 | |
| *** shreshtha has joined #tripleo | 03:54 | |
| *** pdeore has joined #tripleo | 03:55 | |
| *** stendulker has joined #tripleo | 03:57 | |
| *** mdnadeem has joined #tripleo | 03:58 | |
| *** tzumainn has quit IRC | 04:02 | |
| *** ykarel has joined #tripleo | 04:03 | |
| *** mschuppert has joined #tripleo | 04:05 | |
| *** khyr0n has joined #tripleo | 04:06 | |
| *** jtcressy has quit IRC | 04:09 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 04:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 04:10 |
| *** slaweq has joined #tripleo | 04:11 | |
| Tengu | jillr: you're wanting to kill the ci, don't you ;) | 04:12 |
| *** psachin` has quit IRC | 04:15 | |
| *** slaweq has quit IRC | 04:16 | |
| *** psachin` has joined #tripleo | 04:20 | |
| openstackgerrit | Vu Cong Tuan proposed openstack/instack-undercloud master: Switch to stestr https://review.openstack.org/581327 | 04:21 |
| openstackgerrit | Merged openstack/tripleo-common master: Generate rndc key in password list https://review.openstack.org/582366 | 04:22 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Remove notification_driver parameter from heat profile https://review.openstack.org/584270 | 04:22 |
| *** jtcressy has joined #tripleo | 04:24 | |
| *** holser_ has quit IRC | 04:26 | |
| *** itlinux has joined #tripleo | 04:28 | |
| openstackgerrit | Rabi Mishra proposed openstack/puppet-tripleo stable/queens: Remove notification_driver parameter from heat profile https://review.openstack.org/585129 | 04:28 |
| *** holser_ has joined #tripleo | 04:29 | |
| *** Petersingh|afk is now known as Petersingh | 04:30 | |
| *** jtcressy has quit IRC | 04:31 | |
| *** pdeore has quit IRC | 04:32 | |
| *** pfo has quit IRC | 04:34 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: update default logging to match upstream https://review.openstack.org/584088 | 04:39 |
| *** jaganathan has quit IRC | 04:42 | |
| *** jaganathan has joined #tripleo | 04:42 | |
| *** honza has quit IRC | 04:47 | |
| *** honza has joined #tripleo | 04:48 | |
| *** honza is now known as Guest56850 | 04:49 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Remove --use-heat usage, as it's deprecated https://review.openstack.org/581534 | 04:49 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Add --force-stack-create https://review.openstack.org/584356 | 04:49 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Don't assume ansible_dir in finally clause https://review.openstack.org/584367 | 04:49 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Log tracebacks https://review.openstack.org/584434 | 04:49 |
| openstackgerrit | Merged openstack/tripleo-common master: git integration for GetOvercloudConfig action https://review.openstack.org/579634 | 04:49 |
| openstackgerrit | Merged openstack/tripleo-common master: Use /var/lib/mistral/<plan-name> as config-download dir https://review.openstack.org/579635 | 04:49 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Update manila environment file names https://review.openstack.org/583705 | 04:49 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: ansible: replace yum module by package module when possible https://review.openstack.org/584463 | 04:54 |
| openstackgerrit | Merged openstack/tripleo-common master: Add DockerNovaMetadataConfigImage as part of metadata httpd wsgi move https://review.openstack.org/582959 | 04:54 |
| openstackgerrit | Merged openstack/instack-undercloud stable/queens: Configure keepalived before rabbitmq https://review.openstack.org/584852 | 04:54 |
| *** holser_ has quit IRC | 04:56 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-validations master: Enhanced checkdisk for undercloud https://review.openstack.org/584314 | 05:01 |
| *** yprokule has joined #tripleo | 05:02 | |
| Tengu | sorry dpeacock - had to update the patch to reflect directly a future deprecation in ansible -^ ... | 05:03 |
| *** edmondsw has joined #tripleo | 05:03 | |
| *** edmondsw has quit IRC | 05:08 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 05:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 05:10 |
| *** slaweq has joined #tripleo | 05:11 | |
| *** pdeore has joined #tripleo | 05:15 | |
| *** slaweq has quit IRC | 05:15 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-validations master: Enhanced checkdisk for undercloud https://review.openstack.org/584314 | 05:16 |
| *** ratailor has joined #tripleo | 05:19 | |
| *** skramaja has joined #tripleo | 05:21 | |
| openstackgerrit | Janki Chhatbar proposed openstack/tripleo-heat-templates master: Edit some post update tasks logic https://review.openstack.org/579182 | 05:29 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Fix tempest related files permissions for tempest container https://review.openstack.org/584771 | 05:29 |
| *** itlinux has quit IRC | 05:30 | |
| *** cshastri has joined #tripleo | 05:31 | |
| *** quiquell|off is now known as quiquell | 05:38 | |
| *** ccamacho has joined #tripleo | 05:44 | |
| *** ksambor has joined #tripleo | 05:50 | |
| *** pfo has joined #tripleo | 05:52 | |
| *** ffiore has joined #tripleo | 05:57 | |
| *** janki has joined #tripleo | 06:06 | |
| *** Petersingh is now known as Petersingh|afk | 06:07 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 06:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 06:10 |
| *** slaweq has joined #tripleo | 06:11 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Instance create fails due to wrong default secontext with NFS https://review.openstack.org/582913 | 06:12 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates stable/queens: Instance create fails due to wrong default secontext with NFS https://review.openstack.org/582949 | 06:14 |
| *** Petersingh|afk is now known as Petersingh | 06:14 | |
| bandini | mwhahaha, bnemec: argh. /me hides in the corner | 06:16 |
| *** slaweq has quit IRC | 06:16 | |
| Tengu | bandini: hello! care to have another look on https://review.openstack.org/#/c/583886/ and https://review.openstack.org/#/c/584754/ ? That would be great :). | 06:17 |
| *** stendulker_ has joined #tripleo | 06:18 | |
| *** agurenko has joined #tripleo | 06:21 | |
| *** paramite has joined #tripleo | 06:21 | |
| *** stendulker has quit IRC | 06:21 | |
| bandini | ack, I'll try | 06:22 |
| Tengu | thank you :) | 06:23 |
| Tengu | bandini: for info, the overcloud keepalived was also wrong, because it checked on the wrong socket name :). | 06:23 |
| bandini | aha! | 06:23 |
| Tengu | for instance: http://logs.rdoproject.org/54/584754/4/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/6a1ea9f/logs/overcloud-controller-0/etc/haproxy/haproxy.cfg.gz - it's also "stats", not "haproxy.sock" or whatever it was :D | 06:23 |
| bandini | meh | 06:24 |
| Tengu | as you say ;) | 06:24 |
| *** saneax has joined #tripleo | 06:26 | |
| *** pfo has quit IRC | 06:28 | |
| *** Petersingh is now known as Petersingh|afk | 06:30 | |
| *** pcaruana has joined #tripleo | 06:34 | |
| *** Petersingh|afk is now known as Petersingh | 06:35 | |
| *** skramaja_ has joined #tripleo | 06:37 | |
| *** skramaja has quit IRC | 06:37 | |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: Keep plan on update and store user env_files into swift https://review.openstack.org/583145 | 06:38 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: Initial support for adding cluster nodes to an existing cluster https://review.openstack.org/585150 | 06:38 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: DNM Test CI https://review.openstack.org/585151 | 06:39 |
| *** bogdando has joined #tripleo | 06:39 | |
| *** skramaja_ is now known as skramaja | 06:39 | |
| *** yamahata has quit IRC | 06:43 | |
| *** yamahata has joined #tripleo | 06:43 | |
| *** alee has joined #tripleo | 06:45 | |
| *** kopecmartin has joined #tripleo | 06:46 | |
| *** jfrancoa has joined #tripleo | 06:47 | |
| *** jtcressy has joined #tripleo | 06:48 | |
| *** aufi has joined #tripleo | 06:50 | |
| openstackgerrit | Michele Baldessari proposed openstack/instack-undercloud stable/pike: Configure keepalived before rabbitmq https://review.openstack.org/585152 | 06:51 |
| *** quiquell is now known as quiquell|bbl | 06:54 | |
| *** skramaja_ has joined #tripleo | 06:56 | |
| *** skramaja has quit IRC | 06:56 | |
| *** jtcressy has quit IRC | 06:59 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Unify custom t-h-t install steps for UC/OC/upgrade https://review.openstack.org/465047 | 07:00 |
| *** jtcressy has joined #tripleo | 07:02 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates stable/queens: Enable logging to stdout/stderr in memcached https://review.openstack.org/585154 | 07:03 |
| *** stendulker has joined #tripleo | 07:05 | |
| *** udesale has joined #tripleo | 07:06 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates stable/queens: Allow custom --bip CIDR for docker options https://review.openstack.org/584322 | 07:07 |
| *** stendulker_ has quit IRC | 07:08 | |
| *** jtcressy has quit IRC | 07:09 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: New preflight validation: disk space on undercloud https://review.openstack.org/582917 | 07:09 |
| Tengu | bogdando: does the --bip send beep to the local speaker? | 07:10 |
| * Tengu runs away | 07:10 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 07:10 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 07:10 |
| *** cylopez has joined #tripleo | 07:11 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates stable/pike: Allow custom --bip CIDR for docker options https://review.openstack.org/584323 | 07:11 |
| *** slaweq has joined #tripleo | 07:11 | |
| *** slaweq has quit IRC | 07:12 | |
| *** cylopez has left #tripleo | 07:13 | |
| *** amoralej|off is now known as amoralej | 07:13 | |
| *** brault has quit IRC | 07:14 | |
| bogdando | Tengu: eheh | 07:14 |
| *** brault has joined #tripleo | 07:16 | |
| *** slaweq has joined #tripleo | 07:16 | |
| *** tesseract has joined #tripleo | 07:18 | |
| *** dparkes has joined #tripleo | 07:20 | |
| openstackgerrit | Janki Chhatbar proposed openstack/tripleo-heat-templates stable/queens: Edit some post update tasks logic https://review.openstack.org/585159 | 07:24 |
| *** alee has quit IRC | 07:25 | |
| *** alee has joined #tripleo | 07:25 | |
| *** alee has quit IRC | 07:27 | |
| *** alee has joined #tripleo | 07:27 | |
| *** alee has quit IRC | 07:29 | |
| *** dmacpher has quit IRC | 07:29 | |
| *** alee has joined #tripleo | 07:29 | |
| *** gfidente has joined #tripleo | 07:29 | |
| *** gfidente has quit IRC | 07:29 | |
| *** gfidente has joined #tripleo | 07:29 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Don't clean node in ci.centos https://review.openstack.org/584801 | 07:29 |
| *** alee has quit IRC | 07:31 | |
| *** alee has joined #tripleo | 07:31 | |
| *** shardy has joined #tripleo | 07:32 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix for proper LVM support https://review.openstack.org/576168 | 07:32 |
| *** ykarel is now known as ykarel|lunch | 07:33 | |
| *** tosky has joined #tripleo | 07:36 | |
| *** holser_ has joined #tripleo | 07:36 | |
| *** Petersingh is now known as Petersingh|lunch | 07:38 | |
| numans | skramaja_, Hi. do you know how to deploy tripleo quickstart with containerized undercloud ? | 07:39 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: Keep plan on update and store user env_files into swift https://review.openstack.org/583145 | 07:41 |
| *** nyechiel_ has joined #tripleo | 07:42 | |
| *** msufiyan has joined #tripleo | 07:43 | |
| *** msufiyan is now known as msufiyan|lunch | 07:45 | |
| *** khyr0n has quit IRC | 07:45 | |
| *** avivgt has joined #tripleo | 07:45 | |
| *** holser_ has quit IRC | 07:46 | |
| *** nyechiel has quit IRC | 07:47 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Fix tempest related files permissions for tempest container https://review.openstack.org/584771 | 07:48 |
| *** alee has quit IRC | 07:48 | |
| *** holser_ has joined #tripleo | 07:49 | |
| Tengu | hmm, does this error ring bells? http://logs.openstack.org/14/584314/6/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/41f570c/job-output.txt.gz#_2018-07-24_07_07_01_701729 | 07:49 |
| shardy | numans: Hi, I think it's the default now, check https://review.openstack.org/#/c/582405/4/config/general_config/minimal.yml | 07:49 |
| Tengu | chkumar|rover: -^ maybe? | 07:49 |
| numans | shardy, i just started "bash ./quickstart.sh ..." in my setup. I will verify. Thanks. | 07:50 |
| shardy | numans: I think --use-heat is also now not required ref https://github.com/openstack/tripleo-quickstart/commit/596d1177349eed027b939a2fee6d82bbbab69eb5# | 07:50 |
| shardy | numans: ack, I rebased my local config last week and got containers without changing anything fwiw | 07:51 |
| numans | shardy, ok. that's cool. lets see how it goes :) | 07:51 |
| *** quiquell|bbl is now known as quiquell | 07:52 | |
| *** holser_ has quit IRC | 07:52 | |
| chkumar|rover | Tengu: nope, checking with other runs for the same job\ | 07:53 |
| Tengu | chkumar|rover: ok! I'm trying to actually understand what it means | 07:53 |
| *** holser_ has joined #tripleo | 07:55 | |
| numans | shardy, i was using a custom config file for ovn and i noticed that containerized_undercloud is not set to true. so i stopped and enabled containerized_undercloud, ctlplane_masquerade and other related ones. thanks for pointing to the patch | 07:56 |
| shardy | numans: np, hopefully that gets things running for you | 07:56 |
| *** rcernin_ has quit IRC | 07:56 | |
| shardy | also check you have https://github.com/openstack/tripleo-heat-templates/commit/90a7a22f157b0f50d024d730e8b97ba268bc58ad as my undercloud ironic was broken when I deployed it w/containers last week | 07:57 |
| *** pfo has joined #tripleo | 07:57 | |
| chkumar|rover | Tengu: In another jobs that step is getting skipped | 07:57 |
| chkumar|rover | Tengu: from the passed one http://logs.openstack.org/93/583293/6/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/6c33d9d/job-output.txt.gz#_2018-07-24_01_19_02_812589 | 07:58 |
| Tengu | chkumar|rover: hmm. weird. I don't find any log for the step -.-' | 07:58 |
| *** janki has quit IRC | 07:58 | |
| *** janki has joined #tripleo | 07:58 | |
| Tengu | anyway. non-voting, must be some reason for that :) | 07:58 |
| *** suuuper has joined #tripleo | 07:59 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/560445 | 08:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/ocata: GATE CHECK for TripleO https://review.openstack.org/564291 | 08:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/pike: GATE CHECK for TripleO https://review.openstack.org/564285 | 08:00 |
| openstackgerrit | Jaganathan Palanisamy proposed openstack/tripleo-common master: TLS derive parameters for VIPs and FQDNs https://review.openstack.org/582344 | 08:00 |
| jfrancoa | owalsh: hello, do you have a moment for a doubt? | 08:02 |
| *** tcw1 has joined #tripleo | 08:02 | |
| *** stendulker_ has joined #tripleo | 08:03 | |
| *** avivgt has quit IRC | 08:03 | |
| *** tcw has quit IRC | 08:03 | |
| *** stendulker has quit IRC | 08:06 | |
| *** derekh has joined #tripleo | 08:07 | |
| *** jpich has joined #tripleo | 08:07 | |
| *** rcernin_ has joined #tripleo | 08:10 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 08:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 08:10 |
| *** gkadam has joined #tripleo | 08:13 | |
| *** agurenko has quit IRC | 08:18 | |
| *** avivgt has joined #tripleo | 08:18 | |
| *** agurenko has joined #tripleo | 08:18 | |
| *** aufi_ has joined #tripleo | 08:20 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart-extras master: make reproducer bash syntax more portable https://review.openstack.org/581012 | 08:21 |
| *** ykarel|lunch is now known as ykarel | 08:21 | |
| *** aufi has quit IRC | 08:23 | |
| *** maufart__ has joined #tripleo | 08:23 | |
| *** leanderthal has joined #tripleo | 08:23 | |
| *** aufi_ has quit IRC | 08:26 | |
| *** holser_ has quit IRC | 08:27 | |
| *** dmacpher has joined #tripleo | 08:29 | |
| *** holser_ has joined #tripleo | 08:30 | |
| *** aufi_ has joined #tripleo | 08:32 | |
| *** dtantsur|afk is now known as dtantsur | 08:33 | |
| *** jfrancoa has quit IRC | 08:33 | |
| *** Petersingh|lunch is now known as Petersingh | 08:33 | |
| *** florianf has joined #tripleo | 08:33 | |
| *** dtrainor has quit IRC | 08:34 | |
| *** maufart__ has quit IRC | 08:35 | |
| *** akrivoka has joined #tripleo | 08:38 | |
| *** florianf has quit IRC | 08:39 | |
| *** gfidente has quit IRC | 08:46 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 08:46 | |
| *** maufart__ has joined #tripleo | 08:46 | |
| flaper87 | bogdando: is there a way to opt-out from the containerized undercluod? | 08:47 |
| flaper87 | bogdando: also, have you seen ImageDownloadError when using the containerized undercloud? | 08:48 |
| flaper87 | I can't boot an overcloud-full instance as it complains there's no space left to download the image | 08:49 |
| *** aufi_ has quit IRC | 08:49 | |
| *** jfrancoa has joined #tripleo | 08:53 | |
| *** nenad has joined #tripleo | 08:53 | |
| bogdando | flaper87: it's gonna be default as if rocky, but for now we have --use-heat=False to deploy instack | 08:54 |
| bogdando | I know nothing of that ImageDownloadError :( | 08:55 |
| flaper87 | argh, ok | 08:55 |
| flaper87 | I'll try re-installing it | 08:56 |
| *** jfrancoa has quit IRC | 08:57 | |
| dtantsur | flaper87: the default deploy method in rocky can be sensitive to RAM of nodes | 08:58 |
| * dtantsur wonders if we should revert it to not hurt local testing | 08:59 | |
| *** Petersingh is now known as Petersingh|bomga | 08:59 | |
| flaper87 | dtantsur: can you elaborate more? Is that because it downloads images into RAM disks or something? | 09:00 |
| dtantsur | flaper87: precisely | 09:01 |
| *** stendulker has joined #tripleo | 09:01 | |
| dtantsur | flaper87: https://docs.openstack.org/ironic/latest/admin/interfaces/deploy.html#direct-deploy | 09:01 |
| *** janki has quit IRC | 09:01 | |
| *** gouthamr has quit IRC | 09:02 | |
| *** dmellado has quit IRC | 09:03 | |
| *** stevebaker has quit IRC | 09:03 | |
| *** stendulker_ has quit IRC | 09:05 | |
| *** udesale has quit IRC | 09:06 | |
| flaper87 | dtantsur: so, direct is the default one now, right? Which one was used before that switch? | 09:06 |
| dtantsur | flaper87: yes. iscsi. | 09:07 |
| *** Mantorok has quit IRC | 09:08 | |
| *** shreshtha_ has joined #tripleo | 09:08 | |
| *** Mantorok has joined #tripleo | 09:09 | |
| shardy | dtantsur: sounds like we should probably make that opt-in for tripleo, if it significantly increases the undercloud memory requirements? | 09:09 |
| dtantsur | shardy: well, a normal overcloud cannot be installed in 4Gi of RAM, right? | 09:09 |
| shardy | even in the baremetal case folks have run into issues with e.g heat memory consumption so it'd be good to avoid any nasty surprises on upgrade ;) | 09:09 |
| dtantsur | and iscsi scales worse in large deployments | 09:09 |
| *** shreshtha_ has quit IRC | 09:09 | |
| shardy | dtantsur: No, the quickstart default is 12G, 8G can work if you can tolerate some swapping | 09:10 |
| *** shreshtha_ has joined #tripleo | 09:10 | |
| flaper87 | fwiw, my undercluod was deployed with quickstart | 09:10 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 09:10 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 09:10 |
| *** shreshtha has quit IRC | 09:10 | |
| dtantsur | that being said, I'm fine with changing the default back for Rocky, since we're planning some improvements for Stein | 09:10 |
| shardy | dtantsur: Yeah it seems like exposing the option to enabled this, and documenting the advantages would be the safest option for Rocky? | 09:10 |
| flaper87 | ++ | 09:11 |
| shardy | Then maybe in Stein make it the default, but switch back to the less memory intensive option for small e.g quickstart environments? | 09:11 |
| *** shreshtha_ is now known as shreshthaway | 09:11 | |
| dtantsur | shardy: it can be changed by setting IronicDefaultDeployInterface or per node. I'll think about it. | 09:12 |
| flaper87 | dtantsur: setting iscsi now for my env, will test and give you feedback | 09:13 |
| dtantsur | cool! | 09:14 |
| *** jfrancoa has joined #tripleo | 09:15 | |
| *** stendulker_ has joined #tripleo | 09:16 | |
| *** ratailor has quit IRC | 09:18 | |
| *** stendulker has quit IRC | 09:19 | |
| *** Haresh has joined #tripleo | 09:19 | |
| *** ratailor has joined #tripleo | 09:20 | |
| *** panda|off is now known as panda | 09:25 | |
| *** udesale has joined #tripleo | 09:28 | |
| *** dtrainor has joined #tripleo | 09:31 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Fix tempest related files permissions for tempest container https://review.openstack.org/584771 | 09:32 |
| skramaja_ | numans: i dont know if you have got the answer already. but most of the ci is based on undercloud continares today.. by enabling "containerized_undercloud" in the config | 09:33 |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: New preflight validation: disk space on undercloud https://review.openstack.org/582917 | 09:33 |
| Tengu | damn. my brain is still down apparently -.-' | 09:33 |
| numans | skramaja_, i got the answer from shardy. thanks | 09:33 |
| skramaja_ | cool numans | 09:33 |
| *** cylopez has joined #tripleo | 09:34 | |
| *** skramaja_ is now known as skramaja | 09:34 | |
| *** stendulker_ has quit IRC | 09:35 | |
| *** stendulker_ has joined #tripleo | 09:35 | |
| openstackgerrit | Tom Barron proposed openstack/tripleo-heat-templates stable/queens: Update manila environment file names https://review.openstack.org/585190 | 09:36 |
| *** dsneddon has quit IRC | 09:36 | |
| *** shreshthaway has quit IRC | 09:38 | |
| *** shreshtha has joined #tripleo | 09:40 | |
| *** shreshtha has quit IRC | 09:40 | |
| *** shreshtha has joined #tripleo | 09:41 | |
| openstackgerrit | Derek Higgins proposed openstack/tripleo-heat-templates master: Add neutron_plugin_ml2 tag to the ml2 service https://review.openstack.org/585194 | 09:41 |
| *** jaosorior has quit IRC | 09:47 | |
| openstackgerrit | Tom Barron proposed openstack/tripleo-heat-templates master: copy ceph config in manila-share container bundle https://review.openstack.org/584949 | 09:49 |
| *** ratailor has quit IRC | 09:55 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates master: QDR for metrics collection purposes https://review.openstack.org/572312 | 09:55 |
| *** ratailor has joined #tripleo | 09:56 | |
| shardy | flaper87: did IronicDefaultDeployInterface help? I'm still seeing the error I had last week, e.g openstack overcloud node introspect --all-manageable hangs because none of the nodes can pxe boot | 09:58 |
| * shardy wonders why this is working in CI | 09:58 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: WIP Fix spec tests https://review.openstack.org/585197 | 09:58 |
| bogdando | shardy, flaper87: the last time we were facing pxe boot issues for containerized UC and OVB jobs was related to MTU IIRC | 09:59 |
| bogdando | but we're running the switched OVBs now on containerized UC for quite a while... | 10:00 |
| bogdando | we've been* | 10:00 |
| shardy | bogdando: ack, this is a local quickstart run - my config hasn't changed but this is failing repeatedly for me | 10:00 |
| *** avivgt has quit IRC | 10:00 | |
| bogdando | there is also a weird https://bugs.launchpad.net/tripleo/+bug/1782267 no one knows how to debug | 10:00 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 10:00 |
| cylopez | tripleo.org | 10:01 |
| cylopez | down ? | 10:01 |
| bogdando | the last comment shows pxe boot issues as well | 10:01 |
| shardy | bogdando: this is just failing to boot the introspection image, the ipxe stage fails to find any image | 10:01 |
| shardy | so all the nodes just wait with No bootable device. | 10:01 |
| *** zoli is now known as zoli|lunch | 10:01 | |
| bogdando | that's what I'm saying, right | 10:01 |
| bogdando | https://bugs.launchpad.net/tripleo/+bug/1782267/comments/23 'No bootable device' | 10:02 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 10:02 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Fix tempest related files permissions for tempest container https://review.openstack.org/584771 | 10:02 |
| shardy | bogdando: Hmm Ok I'm not seeing any ProcessExcecutionError but perhaps I've not looked at the right logs | 10:02 |
| bogdando | IIRC it was related to dhcp/mtu | 10:02 |
| * shardy still doesn't get why this works in CI | 10:02 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Fix tempest related files permissions for tempest container https://review.openstack.org/584771 | 10:02 |
| bogdando | we introduced mtu for underclouds, set it to 1350 | 10:03 |
| bogdando | for ovb jobs | 10:03 |
| bogdando | made it in parity to instack | 10:03 |
| bogdando | all history is tracked in trello cards... | 10:03 |
| shardy | Ok, my last working undercloud was instack based | 10:03 |
| bogdando | https://trello.com/c/xkyfUGST/34-identify-all-blockers-for-ovb-fs001 | 10:03 |
| bogdando | so there also had been some iptables parity fixes... | 10:04 |
| bogdando | anyway, that worked so we switched OVBs :) | 10:04 |
| *** morazi has quit IRC | 10:04 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-common master: Support for ARA report for ansible playbooks in deploy https://review.openstack.org/565077 | 10:04 |
| shardy | bogdando: Ok but promotion jobs and all local deployments are still broken? | 10:04 |
| jbadiapa | shardy, where is check it on the CI? | 10:05 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: [WIP][DNM] Add ansible module to emit_releases_file.py https://review.openstack.org/585200 | 10:05 |
| bogdando | and more , shardy https://trello.com/c/KoHPlaNd/98-switch-fs035-ovb | 10:05 |
| openstackgerrit | Sagi Shnaidman proposed openstack/python-tripleoclient master: Support ARA report tracking from command line https://review.openstack.org/583799 | 10:05 |
| shardy | jbadiapa: that's what I'm trying to figure out - apparently we switched the OVB jobs and they still work, but locally it's all broken | 10:05 |
| bogdando | introspection timeouts was mentioned in comments | 10:05 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Collect overcloud statistics with ARA https://review.openstack.org/578462 | 10:06 |
| *** betherly-afk is now known as betherly_ | 10:06 | |
| bogdando | shardy: IIRC, promotions jobs had been switched a few months ago https://trello.com/c/3HxQkb0t/4-move-tripleo-ci-centos-7-undercloud-containers-to-gate-and-promotion | 10:06 |
| jbadiapa | sahrdy, what I could see is that is not checked, the prep-images only upload the images but it doesn't do any introspection nor the provide | 10:06 |
| shardy | I tried skipping the introspection and the same failure happens on deploy | 10:07 |
| shardy | bogdando: Ok, tbh I'm more interested in how to fix this than the CI history, but good to have the context | 10:07 |
| shardy | clearly we have a CI gap somewhere as this is completely broken locally | 10:07 |
| *** cylopez has left #tripleo | 10:08 | |
| bogdando | right, just wanted to say it have been working in CI, ther is no 'still broken' things ) | 10:08 |
| bogdando | only 'now broken' ;) | 10:08 |
| shardy | bogdando: have you re-run quickstart recently and it worked? | 10:08 |
| bogdando | the last time I've been debugging OVBs on rdo cloud, like in March or April | 10:09 |
| *** matbu has quit IRC | 10:09 | |
| bogdando | I remember it was before we finished the switching work, and it worked for me | 10:09 |
| shardy | This isn't about OVB or RDO cloud - local deployments via quickstart are broken since the switch to containerized uc | 10:09 |
| shardy | I'm trying to understand why, how to fix, and why we didn't catch it in CI | 10:09 |
| bogdando | I see, sorry, have no data for local deployments | 10:10 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 10:10 |
| * shardy sighs | 10:10 | |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 10:10 |
| bogdando | but I'd start comparing quickstart settings used to deploy vs settings in fs001/35, shardy | 10:10 |
| shardy | I've had the same working config for the last 3 months | 10:10 |
| shardy | that hasn't changed | 10:11 |
| bogdando | hm | 10:11 |
| shardy | so we either have some bad qs defaults, or the containerized undercloud is the cause | 10:11 |
| *** Petersingh|bomga is now known as Petersingh | 10:12 | |
| * shardy tries with --use-heat false | 10:13 | |
| jbadiapa | shardy, this morning I tried to use the non-containerized and the only thing I had to do is to change the "openstack overcloud image upload --http-boot=/httpboot" at the pre-images script | 10:13 |
| bogdando | shardy: I can remember that my local ovb efforts had been constantly suffering from intermittent PXE boot issues :/ | 10:14 |
| jbadiapa | shardy, it worked after a while | 10:14 |
| bogdando | that didn't result into anything, but that's only cuz I'm still relatively new to that HW prov complexity | 10:14 |
| shardy | jbadiapa: ack - was containerized_undercloud: false enough or did you need to add --use-heat=false via undercloud_install_cli_options? | 10:14 |
| shardy | bogdando: Ok, well my local testing has been very reliable until this recent switch to containers | 10:15 |
| jbadiapa | I tried both, but at the end I had to do the provisioning and run the installation manually | 10:15 |
| shardy | which could be a coincidence I guess | 10:15 |
| bogdando | shardy: for sure! | 10:15 |
| bogdando | :) | 10:15 |
| jbadiapa | shardy, openstack undercloud install --use-heat=False | 10:16 |
| shardy | Ok I'll try adding that via undercloud_install_cli_options as I'm not sure how to stop quickstart before it does the install | 10:16 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: WIP: Reproduce CI multinode job with libvirt https://review.openstack.org/543429 | 10:17 |
| *** shreshtha_ has joined #tripleo | 10:19 | |
| *** gfidente has joined #tripleo | 10:19 | |
| *** gfidente has quit IRC | 10:19 | |
| *** gfidente has joined #tripleo | 10:19 | |
| *** shreshtha has quit IRC | 10:19 | |
| *** gouthamr has joined #tripleo | 10:20 | |
| *** shreshtha has joined #tripleo | 10:21 | |
| openstackgerrit | Ricardo Noriega proposed openstack/tripleo-common master: Append Bgpvpn Heat plugin to Neutron Opendaylight server image https://review.openstack.org/585201 | 10:21 |
| *** jaosorior has joined #tripleo | 10:22 | |
| *** shrjoshi has joined #tripleo | 10:22 | |
| *** shreshtha_ has quit IRC | 10:24 | |
| rnoriega | bogdando, ping | 10:24 |
| *** dmellado has joined #tripleo | 10:24 | |
| *** Petersingh is now known as Petersingh|away | 10:25 | |
| *** shreshtha has quit IRC | 10:26 | |
| bogdando | rnoriega: pong | 10:27 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: Remove tripleo.sh --bootstrap-subnodes add ceph loop device https://review.openstack.org/583195 | 10:27 |
| *** msufiyan|lunch has quit IRC | 10:27 | |
| rnoriega | bogdando, hey! thanks for answering my mail about the heat plugin | 10:29 |
| *** msufiyan|lunch has joined #tripleo | 10:29 | |
| rnoriega | bogdando, I'm taking a look at sharing the volumens between containers. | 10:29 |
| rnoriega | bogdando, so in theory, what I should share is the location of the python folder, or the directory where the python components are located (like /usr/lib/python2.7/site-packages..) | 10:30 |
| rnoriega | bogdando, is that right? | 10:30 |
| rnoriega | bogdando, I don't think those folders are mounted in volumes | 10:30 |
| bogdando | rnoriega: you are welcome, go for it. Sadly, the example I brought didn't help to solve that permission denied errors though, even though the hot path permissions with that patch started being chowned properly! ;) | 10:31 |
| bogdando | host path* | 10:31 |
| bogdando | so I think the approach is right | 10:31 |
| bogdando | and for my case, that permission denied error is related to something else | 10:31 |
| bogdando | rnoriega: yes, I think t-h-t is missing that parameter defining plugins dir as a volume mounted in docker containers | 10:32 |
| rnoriega | bogdando, I see... have you ever seen that kind of path mounted in a volume? I don't know if that would have logic in the container world... | 10:32 |
| bogdando | sure, we have tht options for custom volumes | 10:33 |
| bogdando | like http://codesearch.openstack.org/?q=ExtraVolumes&i=nope&files=&repos= | 10:33 |
| *** msufiyan|lunch has quit IRC | 10:34 | |
| bogdando | and http://codesearch.openstack.org/?q=OptVolumes&i=nope&files=&repos=, rnoriega | 10:34 |
| bogdando | so I think that plugin dir may be any dir you want | 10:34 |
| rnoriega | bogdando, let me check | 10:35 |
| bogdando | just make sure it is passed in via some tht param and it's bind mounted in the service yaml | 10:35 |
| *** edmondsw has joined #tripleo | 10:35 | |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Add support for containerized networking-ansible ML2 plugin https://review.openstack.org/585194 | 10:37 |
| *** Petersingh|away has quit IRC | 10:38 | |
| *** stevebaker has joined #tripleo | 10:38 | |
| *** edmondsw has quit IRC | 10:40 | |
| *** alee has joined #tripleo | 10:42 | |
| *** stendulker has joined #tripleo | 10:43 | |
| *** alee has quit IRC | 10:43 | |
| *** alee has joined #tripleo | 10:43 | |
| *** alee has quit IRC | 10:45 | |
| *** alee has joined #tripleo | 10:45 | |
| *** shrjoshi has quit IRC | 10:46 | |
| *** stendulker_ has quit IRC | 10:47 | |
| *** shrjoshi has joined #tripleo | 10:48 | |
| *** quiquell has quit IRC | 10:48 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: [WIP][DNM] Add ansible module to emit_releases_file.py https://review.openstack.org/585200 | 10:49 |
| *** shrjoshi has quit IRC | 10:49 | |
| *** shreshtha_ has joined #tripleo | 10:49 | |
| *** avivgt has joined #tripleo | 10:51 | |
| *** nyechiel_ has quit IRC | 10:51 | |
| *** rcernin_ has quit IRC | 10:52 | |
| *** shrjoshi has joined #tripleo | 10:52 | |
| *** shreshtha has joined #tripleo | 10:55 | |
| *** shrjoshi has quit IRC | 10:55 | |
| *** shreshtha has quit IRC | 10:55 | |
| *** quiquell has joined #tripleo | 10:55 | |
| *** shreshtha_ has quit IRC | 10:55 | |
| *** shreshtha has joined #tripleo | 10:55 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates master: Add reflection of RpcPort to health checks https://review.openstack.org/583629 | 11:00 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: [WIP][DNM] Add ansible module to emit_releases_file.py https://review.openstack.org/585200 | 11:00 |
| *** stendulker_ has joined #tripleo | 11:00 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-common master: Set known hosts when --limit is set. https://review.openstack.org/584994 | 11:01 |
| *** alee has quit IRC | 11:01 | |
| *** stendulker has quit IRC | 11:03 | |
| rnoriega | bogdando, I see something interesting here. In the heat-engine container... I see that neutron package is also installed: /usr/lib/python2.7/site-packages/neutron | 11:03 |
| rnoriega | bogdando, where can I see that definition in tht? | 11:03 |
| *** stendulker_ has quit IRC | 11:04 | |
| *** nyechiel_ has joined #tripleo | 11:05 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-pacemaker master: Fix rspec test on ubuntu os. https://review.openstack.org/585260 | 11:06 |
| chem | bandini: ^ I created a new one, so that if it fails and yours work you still have a workable patch | 11:07 |
| bandini | chem: oh smart man | 11:07 |
| bandini | thanks | 11:07 |
| chem | bandini: locally the spec was working with or without the patch, so can't say if it fixes anything | 11:07 |
| chem | bandini: anyway it seems it doens't break something | 11:08 |
| bandini | chem: aye let's see what CI says | 11:08 |
| *** panda is now known as panda|lunch | 11:10 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 11:10 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 11:10 |
| bogdando | rnoriega: packages installed via http://git.openstack.org/cgit/openstack/tripleo-common/tree/container-images/tripleo_kolla_template_overrides.j2#n76 -like overrides | 11:18 |
| bogdando | tht does not control it, only sets hiera settings et al | 11:18 |
| bogdando | and puppet tags | 11:19 |
| rnoriega | bogdando, that's the thing, I don't see where "heat_engine_packages_append" adds neutron package to heat-engine container | 11:19 |
| rnoriega | bogdando, :-\ | 11:19 |
| *** morazi has joined #tripleo | 11:19 | |
| *** alee has joined #tripleo | 11:24 | |
| openstackgerrit | Merged openstack/diskimage-builder master: block-device lvm: fix umount phase https://review.openstack.org/503958 | 11:26 |
| *** shreshtha has quit IRC | 11:26 | |
| *** udesale has quit IRC | 11:28 | |
| *** atoth has joined #tripleo | 11:29 | |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 11:29 | |
| *** pradk has joined #tripleo | 11:31 | |
| *** zoli|lunch is now known as zoli | 11:31 | |
| bogdando | rnoriega: interesting, I do not know where from it comes, from. Not from kolla, nor from tripleo common :o | 11:31 |
| rnoriega | bogdando, yep.. :-\ | 11:32 |
| bogdando | containers it's a magic | 11:32 |
| rnoriega | bogdando, black magic xD xD | 11:32 |
| etingof | why it can happen that eth1 (local_interface) is not added to br-ctlplane in containerized ucloud? | 11:35 |
| etingof | that ^ causes introspection to stuck | 11:36 |
| bogdando | which net config is used? | 11:36 |
| *** agopi is now known as agopi|brb | 11:36 | |
| bogdando | is it like in https://review.openstack.org/#/c/542556/102/config/general_config/featureset001.yml@36 etingof? | 11:37 |
| *** fultonj has quit IRC | 11:38 | |
| jbadiapa | bogdando, the default one, I did not add any to the deployment | 11:38 |
| *** fultonj has joined #tripleo | 11:38 | |
| openstackgerrit | Ricardo Noriega proposed openstack/tripleo-heat-templates master: [WIP] Sharing BGPVPN Heat plugin volume https://review.openstack.org/585320 | 11:38 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test key https://review.openstack.org/585321 | 11:39 |
| *** wolverineav has joined #tripleo | 11:39 | |
| *** ansmith has quit IRC | 11:39 | |
| jbadiapa | bogdando, not sure if this what you want to see http://paste.openstack.org/show/726513/ | 11:40 |
| bogdando | jbadiapa: this looks like the cause, wrong config | 11:41 |
| bogdando | I think the regression was caused by defaults switched | 11:41 |
| *** agopi|brb has quit IRC | 11:41 | |
| bogdando | w/o adjusted netconfigs in oooq | 11:41 |
| bogdando | EmilienM: ^^ | 11:41 |
| bogdando | the right config for CI should be net-config-undercloud.yaml and not net-config-simple-bridge.yaml | 11:42 |
| jbadiapa | ok, thx | 11:45 |
| *** rh-jelabarre has joined #tripleo | 11:47 | |
| *** amoralej is now known as amoralej|lunch | 11:48 | |
| EmilienM | hello | 11:53 |
| *** dhill_ has joined #tripleo | 11:53 | |
| jaosorior | ẏo! | 11:53 |
| bogdando | jfrancoa: http://logs.openstack.org/15/583515/4/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/cdd567c/job-output.txt.gz#_2018-07-24_09_11_36_725163 :( | 11:53 |
| bogdando | still failing | 11:53 |
| bogdando | jbadiapa: would you update the bug comments please> | 11:54 |
| bogdando | let's fix that promotion feature set to follow https://review.openstack.org/#/c/542556/102/config/general_config/featureset001.yml@36 | 11:54 |
| bogdando | EmilienM: ^^ wrt https://bugs.launchpad.net/tripleo/+bug/1782267 | 11:54 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 11:54 |
| bogdando | EmilienM: it seems we have that opt desynced for non-switched oooq featuresets after the client defaults switched to use heat | 11:55 |
| EmilienM | bogdando: you have a patch? | 11:56 |
| jbadiapa | bogdando, sure but derekh was the one who found the real issue. | 11:56 |
| bogdando | may be we should alter t-h-t defaults at last?.. I forgot why we have historically non working for CI defaults there | 11:56 |
| bogdando | we've had that topic discussed with you and Dan Prince in the past,IIRC | 11:57 |
| *** edmondsw has joined #tripleo | 11:57 | |
| *** pdeore has quit IRC | 11:57 | |
| bogdando | EmilienM: not yet, thinking of it right now. Shall we fix all fs in oooq and leave the productized defaults bad, or fix in tht at last | 11:57 |
| EmilienM | bogdando: you probably know my answer, right? ;-) | 11:58 |
| bogdando | how come promotion job has shttp://paste.openstack.org/show/726513/ | 11:58 |
| bogdando | jbadiapa: that's for the promotion job, right? | 11:58 |
| bogdando | EmilienM: I think so) | 11:58 |
| EmilienM | bogdando: go fix the bad defaults | 11:58 |
| bogdando | ok, let's back to that concern for tht defaults once again ... ) | 11:58 |
| bogdando | will do thepatch | 11:59 |
| jbadiapa | bogdando, I used the --release master-tripleo-ci | 12:00 |
| bogdando | EmilienM: hmmm, it's there already https://review.openstack.org/#/c/352037/24/environments/undercloud.yaml@4 | 12:01 |
| bogdando | so that's something bad configured in oooq | 12:01 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Upgrade: Ensure host's haproxy service has been stopped https://review.openstack.org/585328 | 12:02 |
| jaosorior | EmilienM, Tengu ^^ that should do it | 12:02 |
| jaosorior | yprokule: got an environment to test that? ^^ | 12:02 |
| EmilienM | jaosorior: we have a CI job | 12:02 |
| EmilienM | (I know that sounds crazy) | 12:02 |
| bandini | chem: seems your patch is fine | 12:03 |
| jaosorior | EmilienM: The issue was seen when TLS is enabled. | 12:03 |
| EmilienM | jaosorior: yep | 12:03 |
| EmilienM | and it's enabled in our CI job | 12:03 |
| EmilienM | I know it sounds crazy! | 12:03 |
| jaosorior | haha | 12:03 |
| EmilienM | jaosorior: let me show you, one sec | 12:03 |
| jaosorior | EmilienM: it's strange then that this is not showing up in CI | 12:03 |
| Tengu | jaosorior: will check that | 12:03 |
| EmilienM | jaosorior: http://logs.openstack.org/38/583238/3/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/46de549/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz | 12:04 |
| Tengu | jaosorior: you might want to set the "enabled: false" in order to unload the service | 12:04 |
| EmilienM | jaosorior: your code isn't enough | 12:04 |
| jaosorior | EmilienM: seems it's not using TLS (the job you showed me) | 12:05 |
| yprokule | EmilienM: do we actually perform upgrade from non-containerized to containerized uc ? | 12:05 |
| EmilienM | yprokule: yes we do | 12:05 |
| Tengu | yprokule: yep, for queens -> rocky | 12:05 |
| EmilienM | jaosorior: it should use TLS since it's default | 12:06 |
| jaosorior | EmilienM: there are some jobs that remove it explicitly | 12:06 |
| jaosorior | this might be the case | 12:06 |
| EmilienM | anyway | 12:06 |
| jaosorior | (but it isn't using it, I checked) | 12:06 |
| EmilienM | jaosorior: ok, let me look that now | 12:06 |
| EmilienM | jaosorior: anyway for the upgrade tasks, please look how we upgrade docker/nova-api | 12:07 |
| jaosorior | EmilienM: oh, and why isn't that enough? is it the "enabled" bit that Tengu mentioned that's missing? | 12:07 |
| jaosorior | ok | 12:07 |
| Tengu | jaosorior: add "enabled: no" so that it will unload the service from the reboot. | 12:07 |
| bogdando | EmilienM: I think all these now broken after the client defaults switched from instack , http://codesearch.openstack.org/?q=ci%2Fcommon%2Fnet-config-simple-bridge.yaml&i=nope&files=&repos= | 12:07 |
| Tengu | else, you might get a running HAProxy on the host after a simple reboot :). | 12:07 |
| yprokule | EmilienM: Tengu how it works ? | 12:07 |
| yprokule | EmilienM: Tengu e.g https://review.openstack.org/#/c/584286/ | 12:08 |
| bogdando | with the current netconfig that eth1 (local_interface) is not added to br-ctlplane in containerized ucloud | 12:08 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Add support for containerized networking-ansible ML2 plugin https://review.openstack.org/585194 | 12:08 |
| *** yrabl has joined #tripleo | 12:09 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Take environment_type out of TOCI_JOBTYPE https://review.openstack.org/582385 | 12:09 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Take nodes out of TOCI_JOBTYPE https://review.openstack.org/582386 | 12:09 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Upgrade: Ensure host's haproxy service has been stopped https://review.openstack.org/585328 | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 12:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 12:10 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-upgrade master: cont/uc: remove --use-heat https://review.openstack.org/585330 | 12:10 |
| bogdando | oh wait, EmilienM, I think the situation is more complicated as we had to use different netconfigs for multinode jobs (to have vxlan magic) vs ovbs | 12:10 |
| EmilienM | yprokule: one sec plz I have 10 pings at the same time... | 12:10 |
| bogdando | so I'm really not sure what to do | 12:10 |
| bogdando | may be we should only try to change it for the faily promotion job? | 12:10 |
| jaosorior | EmilienM, Tengu: uploaded another patch. I gotta pair another bug now with a colleague though :/ so if something's missing feel free to modify that patch. Else, I'll continue it tomorrow. | 12:11 |
| bogdando | which fs is it, folks? in https://bugs.launchpad.net/tripleo/+bug/1782267 ? | 12:11 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 12:11 |
| *** trown|outtypewww is now known as trown | 12:11 | |
| Tengu | grmbl. | 12:11 |
| Tengu | had a network glitch. | 12:11 |
| Tengu | jaosorior: guess your patch didn't make it to gerrit. | 12:12 |
| Tengu | ah, yes, it did. ok. | 12:13 |
| *** strigazi has quit IRC | 12:13 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs050: enable TLS https://review.openstack.org/585333 | 12:13 |
| EmilienM | jaosorior: ^ it was missing | 12:13 |
| EmilienM | jaosorior: maybe you can depends-on this one | 12:14 |
| *** strigazi has joined #tripleo | 12:14 | |
| EmilienM | and the patch isn't good enough... I'll take over | 12:14 |
| EmilienM | it doesn't support the cleanup case | 12:14 |
| jaosorior | EmilienM: thanks for checking it out. I can pick it up in some hours, or tomorrow. | 12:15 |
| EmilienM | i'll take it now | 12:15 |
| jaosorior | thanks | 12:15 |
| EmilienM | jaosorior, Tengu : and keepalived? is it ok? | 12:15 |
| *** nyechiel_ has quit IRC | 12:15 | |
| Tengu | EmilienM: still waiting for the vrrp correction to merge, if this is your question | 12:15 |
| Tengu | btw, PTAL :) https://review.openstack.org/#/c/583886/ https://review.openstack.org/#/c/584754/ - Thank you folks! | 12:16 |
| EmilienM | Tengu: no I'm asking if upgrade worked for this service | 12:16 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Fix tempest related files permissions for tempest container https://review.openstack.org/584771 | 12:16 |
| yprokule | jaosorior: I'll try to re-run with patch applied, though not sure if it'll work | 12:17 |
| Tengu | EmilienM: don't know, I just jumped in with what looked like a possible TLS issue, and ended up being the host HAProxy thingy. I did not get issues on my test. | 12:17 |
| Tengu | at least services were running, but I didn't run "openstack" commands. | 12:17 |
| EmilienM | Tengu: I guess I'm asking if you had keepalived running after the upgrade, outside of the container | 12:18 |
| EmilienM | but i'll give it a try | 12:18 |
| Tengu | EmilienM: ah! hmm didn't check that specific point, sorry | 12:18 |
| *** ratailor has quit IRC | 12:19 | |
| bogdando | shardy: which netcofig you use for your env with pxe timeouts> | 12:19 |
| bogdando | is it tht default for UC? | 12:19 |
| bogdando | do you have eth1 added to br-ctlplane? | 12:20 |
| bogdando | for the context https://bugs.launchpad.net/tripleo/+bug/1782267/comments/25 | 12:20 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 12:20 |
| Tengu | EmilienM: running a build - should be up in a couple of minutes. | 12:20 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Add upgrade_tasks for HAproxy https://review.openstack.org/585328 | 12:22 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs050: enable TLS https://review.openstack.org/585333 | 12:22 |
| EmilienM | jaosorior, Tengu ^ that should 1) Add correct and complete upgrade_tasks for HAproxy and 2) Enable TLS on containerized undercloud upgrade job | 12:22 |
| Tengu | EmilienM: why not use service_fact? | 12:23 |
| EmilienM | and yeah we have upgrade_tasks for keepalived, so it's all good | 12:23 |
| EmilienM | Tengu: set_fact? | 12:23 |
| Tengu | nope | 12:23 |
| Tengu | 2s | 12:23 |
| Tengu | https://docs.ansible.com/ansible/latest/modules/service_facts_module.html#service-facts-module | 12:24 |
| Tengu | this avoids using "command" in order to get a service status. | 12:24 |
| EmilienM | "This module is flagged as preview which means that it is not guaranteed to have a backwards compatible interface." | 12:24 |
| EmilienM | I guess it's a good reason to not using it now | 12:25 |
| Tengu | ok... :( | 12:25 |
| EmilienM | it's new in 2.5 and we just use 2.5 for some weeks now | 12:25 |
| bogdando | long story short https://bugs.launchpad.net/tripleo/+bug/1782267/comments/27 | 12:25 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 12:25 |
| EmilienM | I guess we can start using it | 12:25 |
| shardy | bogdando: it's the default | 12:25 |
| Tengu | funky. I was sure I was using it in my tripleo-lab. | 12:26 |
| bogdando | shardy: do you have eth1 in br-ctlplane> | 12:26 |
| bogdando | can you also post the undercloud-parameter-defaults.yaml ? | 12:26 |
| Tengu | EmilienM: fact is, I tend to avoid using "command" if a module does what I need. | 12:26 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/tripleo-ci master: check file existance and permission for a nodepool private key https://review.openstack.org/585336 | 12:27 |
| Tengu | EmilienM: but its doc is lacky - had to output the content of the hash in order to find the wanted values. | 12:27 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Fix tempest related files permissions for tempest container https://review.openstack.org/584771 | 12:28 |
| *** psachin` has quit IRC | 12:30 | |
| shardy | bogdando: I'll have to recreate to find out, I tried with --use-heat false and that fails because overcloud-prep-images.yml can't find /var/lib/ironic/httpboot/agent.kernel | 12:30 |
| shardy | so it seems non containerized quickstart is also broken | 12:31 |
| *** nyechiel_ has joined #tripleo | 12:31 | |
| *** pchavva has joined #tripleo | 12:32 | |
| bogdando | shardy: yeah, we have a note for that https://docs.openstack.org/tripleo-docs/latest/install/basic_deployment/basic_deployment_cli.html#upload-images :) | 12:34 |
| *** avivgt has quit IRC | 12:34 | |
| bogdando | jfrancoa: bad luck for the patch! it's blocked with https://bugs.launchpad.net/bugs/1783303 :) | 12:35 |
| openstack | Launchpad bug 1783303 in tripleo "[master] scenario000 multinode upgrade job is failing at tripleo-upgrade include task giving controllers' is undefined"}" [Critical,Triaged] | 12:35 |
| *** tzumainn has joined #tripleo | 12:35 | |
| bandini | EmilienM: https://review.openstack.org/#/c/585260/ needed CI fix if you have a min | 12:35 |
| *** panda|lunch is now known as panda | 12:36 | |
| bogdando | so we can't test https://review.openstack.org/#/c/465047/ unless that fixed | 12:36 |
| EmilienM | bandini: k | 12:37 |
| shardy | bogdando: Ok so that's a quickstart bug, as containerized_undercloud: false doesn't select the old path | 12:37 |
| bandini | merci | 12:37 |
| *** rlandy has joined #tripleo | 12:39 | |
| bogdando | shardy: I can't tell for sure, unless I can see eth1 is added in br-ctlplane with t-h-t defaults for UC netconfig | 12:39 |
| jfrancoa | bogdando: auch...a new one. Will have a look at it | 12:39 |
| bogdando | but I think it is, as OVBs are not affected... | 12:40 |
| shardy | bogdando: do we have any CI coverage now for the baremetal instack-undercloud ? | 12:40 |
| bogdando | EmilienM: I think we have only fs003 left? | 12:40 |
| *** leanderthal has quit IRC | 12:41 | |
| bogdando | and fs051 (for technical reasons) | 12:41 |
| EmilienM | bogdando: yes both of them | 12:41 |
| EmilienM | that's it | 12:41 |
| EmilienM | bogdando: wait are you going to patch oooq finally? I thought you would fix THT | 12:41 |
| bogdando | there is also jobs mentioned by Wes in https://bugs.launchpad.net/tripleo/+bug/1782267/comments/5 but I know nothing of those | 12:42 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 12:42 |
| bogdando | and which configs they use | 12:42 |
| bogdando | EmilienM: nothing to fix in tht, see above ) | 12:42 |
| bogdando | the summary in the last two comments in the bug | 12:43 |
| *** avivgt has joined #tripleo | 12:49 | |
| shardy | So the problem is I'm missing this revert | 12:50 |
| shardy | https://github.com/openstack/python-tripleoclient/commit/9ac6bd66af06bfb015a4f903a412b0eadee9e153#diff-84904835afd9ead54ffafbffda5079f6 | 12:50 |
| shardy | quickstart gets python-tripleoclient-10.3.1-0.20180724051058.a33b149.el7.noarch | 12:50 |
| shardy | despite using --release master-tripleo-ci | 12:51 |
| *** pdeore has joined #tripleo | 12:51 | |
| *** ansmith has joined #tripleo | 12:52 | |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: Remove tripleo.sh --bootstrap-subnodes add ceph loop device https://review.openstack.org/583195 | 12:52 |
| shardy | Hmm that should be HEAD | 12:53 |
| shardy | https://github.com/openstack/python-tripleoclient/commit/1e2af1aeb31be7168780d2bc053210a94f5fcc55#diff-84904835afd9ead54ffafbffda5079f6 | 12:55 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: WIP Set default number of rabbitmq to CEIL(N/2) https://review.openstack.org/585340 | 12:55 |
| openstackgerrit | Merged openstack/puppet-pacemaker master: Fix rspec test on ubuntu os. https://review.openstack.org/585260 | 12:55 |
| shardy | bogdando: so merging that revert-revert appears to break baremetal undercloud | 12:55 |
| shardy | because it's still looking at the IRONIC_HTTP_BOOT_BIND_MOUNT dir, which only works for containers | 12:55 |
| bogdando | we are not supposed to deploy instack in rocky any more, hence switched | 12:56 |
| bogdando | EmilienM: ^^ | 12:56 |
| shardy | bogdando: it's deprecated, we shouldn't intentionally break it! | 12:56 |
| shardy | And since the containerized uc is broken for me, I want to switch back to it :( | 12:56 |
| *** mcornea has joined #tripleo | 12:57 | |
| bogdando | the alternative fix for instack has been declined | 12:57 |
| * shardy wasted all day trying to get a working uc | 12:57 | |
| bogdando | I'm not sure how to proceed with that, feel free to revert it again | 12:57 |
| EmilienM | bogdando: agree with shardy - we won't break it | 12:57 |
| EmilienM | shardy: what needs to be reverted? | 12:57 |
| EmilienM | sorry if we broke something | 12:58 |
| EmilienM | oh this httpboot thing | 12:58 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: Initial support for adding cluster nodes to an existing cluster https://review.openstack.org/585150 | 12:58 |
| bogdando | https://review.openstack.org/#/c/564162, EmilienM, shardy, dtantsur | 12:58 |
| bogdando | that was the fix intended to make the swithc non breaking for instack BM deployments | 12:59 |
| *** skramaja has quit IRC | 12:59 | |
| shardy | EmilienM: yeah https://review.openstack.org/#/c/564096/ landed, then we reverted the revert in I2867598f717b3126071e77a7826f48f6c7584ce2 | 12:59 |
| shardy | perhaps we can work around it in quickstart by modifying the --http-boot path when containerized_undercloud: false | 12:59 |
| EmilienM | shardy: https://review.openstack.org/#/c/584265/ didn't help? | 12:59 |
| shardy | e.g the opposite of what we used to do | 12:59 |
| EmilienM | probably unrelated but still wondering | 12:59 |
| EmilienM | shardy: yes it would be a valid workaround to me | 13:00 |
| shardy | EmilienM: No, I still can't get any nodes to pxe boot with containerized uc | 13:00 |
| shardy | so wanted to switch back temporarily to instack as I've got stuff I need to test that doesn't care about containers | 13:00 |
| EmilienM | shardy: what featureset are you using? | 13:00 |
| EmilienM | shardy: go ahead! | 13:00 |
| shardy | EmilienM: it's a local config, pretty much the defaults | 13:00 |
| EmilienM | --use-heat=False | 13:01 |
| shardy | EmilienM: ack thanks will do | 13:01 |
| EmilienM | just don't tell anyone we did that | 13:01 |
| shardy | yeah I set containerized_undercloud: false | 13:01 |
| shardy | undercloud_install_cli_options: " --use-heat false" | 13:01 |
| shardy | but that fails in a different way (tm) ;) | 13:01 |
| shardy | I'll push a t-q-e patch, thanks! | 13:01 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/instack-undercloud master: Alter default http boot path for containerized Ironic https://review.openstack.org/564162 | 13:01 |
| *** pradk has quit IRC | 13:04 | |
| EmilienM | shardy: let us know | 13:06 |
| Tengu | EmilienM: care to have a look at https://review.openstack.org/#/c/583886/ if you have a couple of minutes? :) | 13:09 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-common master: Support for ARA report for ansible playbooks in deploy https://review.openstack.org/565077 | 13:09 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-quickstart-extras master: Switch httpboot directory for baremetal underclouds https://review.openstack.org/585344 | 13:09 |
| shardy | EmilienM: ^^ pushed that but just testing it now, feedback welcome | 13:10 |
| shardy | seems like a simple interim fix | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 13:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 13:10 |
| EmilienM | shardy: looking | 13:10 |
| *** cdearborn has joined #tripleo | 13:11 | |
| *** leanderthal has joined #tripleo | 13:14 | |
| chkumar|rover | mwhahaha: Hello | 13:15 |
| mwhahaha | chkumar|rover: hi2u | 13:15 |
| *** agopi|brb has joined #tripleo | 13:15 | |
| *** agopi|brb is now known as agopi | 13:15 | |
| chkumar|rover | mwhahaha: How can i pull ceph container so that it will be available on undercloud local registery https://review.openstack.org/#/c/581607/ ? | 13:16 |
| EmilienM | (we really need more doc around container image prepare) | 13:16 |
| mwhahaha | chkumar|rover: I'm not sure we pre-fetch ceph during the undercloud install because we don't use it | 13:17 |
| EmilienM | no we don't prefetch | 13:17 |
| mwhahaha | chkumar|rover: so i don't think you can do that unless we prefetch all the containers which is what we were trying to do with the pre run changes to fetch all the containers | 13:17 |
| EmilienM | stevebaker updated the doc patch for "openstack tripleo container image prepare", nice https://review.openstack.org/#/c/553104/ | 13:17 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/instack-undercloud master: Alter default http boot path for BM Ironic https://review.openstack.org/564162 | 13:22 |
| bogdando | shardy, dtantsur: I hope that works ^^ | 13:22 |
| bogdando | EmilienM, mwhahaha: ^^ | 13:22 |
| dtantsur | bogdando: what is going to do the upgrade then? | 13:22 |
| bogdando | ...impact? | 13:22 |
| dtantsur | I assume you don't want to re-run 'image upload' | 13:22 |
| dtantsur | bogdando: you're going to end up with empty /httpboot (without images) | 13:23 |
| bogdando | we do not support upgrading instack into instack | 13:23 |
| openstackgerrit | Martin André proposed openstack/tripleo-common master: Add container images for CNS https://review.openstack.org/582609 | 13:23 |
| openstackgerrit | Martin André proposed openstack/tripleo-common master: Add ability to filter container images to modify https://review.openstack.org/579918 | 13:23 |
| bogdando | that's the trend as if Rocky | 13:23 |
| bogdando | dtantsur: there is upgrade tasks in tht | 13:23 |
| dtantsur | bogdando: then why do you care about u-c at all? just document running 'image upload' with the right args | 13:23 |
| bogdando | dtantsur: I think the idea is to not break this http://codesearch.openstack.org/?q=IRONIC_HTTP_BOOT_BIND_MOUNT&i=nope&files=&repos= | 13:24 |
| *** agopi is now known as agopi|brb | 13:25 | |
| dtantsur | bogdando: I just feel a bit weird about supporting instack-undercloud in rocky, but not supported upgrades to it.. | 13:25 |
| dtantsur | I would think the other way around: we can support upgrades, but please don't do new deployments | 13:25 |
| bogdando | EmilienM, shardy, dtantsur: that's the goal :) | 13:25 |
| dtantsur | the goal is having people make new deployments? which may not be even upgradable to stein? | 13:25 |
| bogdando | not supporting, but not breaking instack-undercloud in rocky | 13:26 |
| bogdando | that's the subject of that pain story above | 13:26 |
| EmilienM | mwhahaha: I can't attend tripleo meeting today, but 2 things : 1/ I'll propose rocky-m3 on Thursday and 2/ I'll propose a stable release of ocata/pike/queens next week (probably Monday). I'll send the Weekly Owl this afternoon. Also this week we need to handle open blueprints. Close the ones that we consider done, defer the one without FFE and postpone the ones with FFE to rocky-rc1 (we also need to create | 13:26 |
| EmilienM | rocky-rc1 in launchpad) | 13:26 |
| dtantsur | bogdando: my definition of "breaking" includes breaking upgrades. but I won't -1 that I guess. | 13:26 |
| mwhahaha | EmilienM: k | 13:26 |
| *** eck`gone is now known as eck` | 13:26 | |
| shardy | dtantsur: I think the plan is to make the supported upgrade path only to the new containerized undercloud, but sometimes for testing it's nice to do baremetal things, which currently are broken via quickstart | 13:27 |
| bogdando | instack to containers upgrades won't be breaking | 13:27 |
| *** agurenko has quit IRC | 13:27 | |
| bogdando | dtantsur: , /thhroot contents migrated via tht upgrade tasks | 13:27 |
| shardy | I pushed a t-q-e patch which may be enough, basically switch the args back in the baremetal case | 13:27 |
| dtantsur | bogdando: yeah, I meants i-u to i-u | 13:28 |
| bogdando | as of instack to instack - yes, that's the call, we do not support it as if rocky... | 13:28 |
| dtantsur | shardy: "for testing it's nice to do baremetal things" worries me | 13:28 |
| dtantsur | I guess it means that it's still much harder to use containerized undercloud for development | 13:28 |
| shardy | dtantsur: pxeboot is broken for me atm, I'm trying to figure out if it's due to the recent switch to containerized uc or not | 13:28 |
| *** medberry has joined #tripleo | 13:28 | |
| *** medberry has joined #tripleo | 13:28 | |
| dtantsur | shardy: during introspection or deployment? | 13:29 |
| *** tcw1 has quit IRC | 13:29 | |
| bogdando | that's another story, just explaining the case | 13:29 |
| openstackgerrit | Merged openstack/tripleo-docs master: Replace port 35357 with 5000 for "auth_url" https://review.openstack.org/569693 | 13:29 |
| bogdando | why we want keep it non breaking for rocky w/o upgrade promises | 13:29 |
| shardy | dtantsur: both, but quickstart hangs up trying to introspect the nodes | 13:29 |
| *** tcw has joined #tripleo | 13:29 | |
| *** agopi|brb has quit IRC | 13:29 | |
| shardy | I set the state of them manually and the deploy also failed | 13:29 |
| dtantsur | bogdando: fair enough | 13:29 |
| bogdando | think of it as the last breath L)( | 13:29 |
| bogdando | Last Breath of Instack | 13:29 |
| dtantsur | lol | 13:30 |
| shardy | I wanted to switch back to instack because that's been working fine for me over the last 3 months | 13:30 |
| dtantsur | bogdando: okay, I'll wait for the CI before voting on it | 13:30 |
| * shardy should have snapshotted his working uc :( | 13:30 | |
| shardy | and also to bisect the problem | 13:30 |
| mwhahaha | thought there was an outstanding issue around introspection bits | 13:30 |
| dtantsur | shardy: I feel like we should start running a CI job with libvirt+quickstart | 13:30 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Fix deploy health checks https://review.openstack.org/584119 | 13:30 |
| shardy | dtantsur: yeah we seem to have lost that coverage along the way | 13:30 |
| shardy | but I'm still not clear why this introspection/pxe issue isn't manifesting in CI failures | 13:31 |
| mwhahaha | shardy: are you running into http://bugs.launchpad.net/bugs/1782267 | 13:31 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 13:31 |
| *** agurenko has joined #tripleo | 13:31 | |
| shardy | mwhahaha: the symptoms are the same, e.g the ipxe timeout on the node console | 13:31 |
| *** toure is now known as toure|brb | 13:31 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/tripleo-ci master: check file existance and permission for a nodepool private key https://review.openstack.org/585336 | 13:31 |
| shardy | mwhahaha: but I dont' see that ProcessExecutionError | 13:31 |
| *** agopi|brb has joined #tripleo | 13:31 | |
| *** lblanchard has joined #tripleo | 13:32 | |
| shardy | mwhahaha: I didn't see a workaround in the bug so decided to try reverting to baremetal so I can make progress | 13:32 |
| mwhahaha | shardy: the issue was in libvirt i think https://bugzilla.redhat.com/show_bug.cgi?id=1576464 | 13:32 |
| openstack | bugzilla.redhat.com bug 1576464 in libvirt "Hash operation not allowed during iteration" [High,Verified] - Assigned to mprivozn | 13:32 |
| dtantsur | that bug should not depend on containers vs baremetal, I think | 13:32 |
| shardy | mwhahaha: ack thanks, there's a few different issues referenced in the lp bug so I wasn't quite sure on the rca | 13:33 |
| *** tcw1 has joined #tripleo | 13:34 | |
| *** tcw has quit IRC | 13:34 | |
| *** aufi_ has joined #tripleo | 13:34 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Update the config for FS021 https://review.openstack.org/583202 | 13:34 |
| *** tcw1 is now known as tcw | 13:35 | |
| bogdando | > shardy, but I'm still not clear why this introspection/pxe issue isn't manifesting in CI failures | 13:35 |
| bogdando | see the bug comments explaing that | 13:36 |
| bogdando | https://bugs.launchpad.net/tripleo/+bug/1782267 | 13:36 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 13:36 |
| *** amoralej|lunch is now known as amoralej | 13:36 | |
| bogdando | it isn't manifesting in CI failures cuz of the right netconfig used | 13:36 |
| bogdando | at least for OVB jobs... | 13:37 |
| *** maufart__ has quit IRC | 13:37 | |
| bogdando | and non-ovb multi-node jobs use another net config as it installs some CI-specific VXLAN magic | 13:37 |
| bogdando | but that's also another story >< | 13:37 |
| bogdando | so it seems we faced the case we want both bosses in da house | 13:38 |
| *** dprince has joined #tripleo | 13:38 | |
| bogdando | vxlan magic and BM provisioning supported :) (no, I hope we don't) | 13:38 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/net-config-undercloud.j2.yaml | 13:39 |
| bogdando | or we need some improvements for netconfigs in ci/commons | 13:39 |
| *** maufart__ has joined #tripleo | 13:39 | |
| shardy | bogdando: so do we need to fix that default config for non-ci users, or just get the libvirt fix? | 13:39 |
| bogdando | hard to tell, really | 13:39 |
| *** bdodd has quit IRC | 13:40 | |
| bogdando | all I know is some CI jobs (multinode) use https://github.com/openstack/tripleo-heat-templates/blob/master/ci/common/net-config-simple-bridge.yaml | 13:41 |
| bogdando | and that can't work fine with Ironic | 13:41 |
| bogdando | as it won't add eth1 into br-ctlplane | 13:41 |
| *** aufi_ has quit IRC | 13:42 | |
| bogdando | as far as we do not expect Ironic in multi-node jobs, that works | 13:42 |
| bogdando | I'm not sure why we have that "conflict" arised for the subj bug | 13:42 |
| *** pdeore has quit IRC | 13:42 | |
| shardy | There's a bunch of different data in the bug but no root-cause-analusis | 13:42 |
| shardy | so it's confusing | 13:43 |
| bogdando | the last comments provide some RCA insights I hope | 13:43 |
| shardy | Ok my t-q-e patch worked, quickstart ran to completion | 13:43 |
| *** ccamacho1 has joined #tripleo | 13:43 | |
| shardy | so it does seem at least partially related to containers | 13:43 |
| *** ccamacho has quit IRC | 13:43 | |
| bogdando | no interface added to the bridge, doesn't that sound as RC? | 13:43 |
| bogdando | shardy: to net configs adopted with tht | 13:44 |
| bogdando | as a side car feature of containers adoption | 13:44 |
| *** avivgt has quit IRC | 13:44 | |
| bogdando | before that, instack was happy riunning os-net-config overrides AFAICT | 13:44 |
| shardy | yeah I'm asking is the config in t-h-t broken, since it's not tested in CI, and reverting to the instack configured networking appears to fix the issue | 13:45 |
| shardy | you reference the config in your comment, so I was wondering if you'd figured that out | 13:45 |
| bogdando | btw, os-net-config overrides are also supported in tht | 13:45 |
| bogdando | see net_config_override in undercloud.conf | 13:45 |
| *** bdodd has joined #tripleo | 13:45 | |
| shardy | ack, my point is the quickstart defaults don't seem to be tested in CI due to the OVB specific configuration | 13:46 |
| bogdando | shardy: just posted summary of jbadiapa and derekh findings | 13:46 |
| shardy | bogdando: Ok thanks | 13:46 |
| bogdando | and my thoughts about bad netconfig as the root cause | 13:46 |
| owalsh | jfrancoa: just had a thought... do we only need to support containers for the ssh known hosts changes? | 13:46 |
| bogdando | 4:45:04 PM GMT+3 - shardy: yeah I'm asking is the config in t-h-t broken, since it's not tested in CI […] | 13:47 |
| bogdando | well, the tht default undercloud netconfig is tested in OVB | 13:47 |
| bogdando | other choice undre test is that net simple bridge config, but limited to multi-node cases | 13:48 |
| jfrancoa | owalsh: hey, sorry I don't follow... what do you mean by supporting only containers? | 13:48 |
| bogdando | no more net configs being tested with UC so far | 13:48 |
| owalsh | jfrancoa: we've dropped support for deploying to baremetal now, yes? | 13:48 |
| bogdando | I did manual tests for net config overrides to parity instack | 13:48 |
| bogdando | WFM :) | 13:48 |
| jfrancoa | owalsh: right | 13:48 |
| bogdando | shardy: && | 13:49 |
| shardy | Ok, but either way the defaults don't work on a local baremetal host atm | 13:49 |
| bogdando | yeah, that is not tested as you folks mentioned above | 13:49 |
| bogdando | libvirt and ironic | 13:49 |
| owalsh | jfrancoa: ok... so we don't really need the ssh known host setup for the sshd on baremetal.... | 13:49 |
| jfrancoa | owalsh: afik, the deployment by default now is with containers | 13:49 |
| owalsh | jfrancoa: need it setup in the nova_migration_target container sshd, which right now inherits the host key from the baremetal host | 13:49 |
| jfrancoa | owalsh: then, we could drop that role running step? | 13:50 |
| *** shardy has quit IRC | 13:50 | |
| owalsh | jfrancoa: yea, unless something else depends on this now | 13:50 |
| jfrancoa | owalsh: I am not really sure about that, could you please comment on the review https://review.openstack.org/#/c/584994/ ? | 13:52 |
| owalsh | jfrancoa: sure | 13:52 |
| jfrancoa | owalsh: I also wanted to comment with my last change that it doesn't make too much sense for me to have a whole role to run a simple task | 13:52 |
| dtantsur | folks, if I want to override some THT parameters when installing the new undercloud, how do I do it? | 13:53 |
| dtantsur | bogdando: ^^^ | 13:53 |
| *** bogdando has quit IRC | 13:54 | |
| *** bdodd has quit IRC | 13:54 | |
| jfrancoa | owalsh: by the way, if we remove the call to tripleo-ssh-known-hosts (from deploy-steps), do you think it would make sense to still leave the role in tripleo-common? I'll do a quick search if we invoke the role from somewhere else | 13:54 |
| *** quiquell has quit IRC | 13:54 | |
| owalsh | slagle: ^^^ might know, I'm assume this is only really required for migration on the computes (that's why it was originally added at least) | 13:55 |
| *** toure|brb is now known as toure | 13:56 | |
| *** shardy has joined #tripleo | 13:56 | |
| jfrancoa | owalsh: good, thanks for the context and the help | 13:56 |
| owalsh | jfrancoa: np... this is a fun one :-) | 13:56 |
| *** avivgt has joined #tripleo | 13:57 | |
| openstackgerrit | Jiri Stransky proposed openstack/python-tripleoclient master: Remove parameter for ceph-ansible playbook from update/upgrade prepare https://review.openstack.org/585366 | 13:58 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-common master: Add default for ceph_ansible_playbook in update/upgrade prepare workflow https://review.openstack.org/585367 | 13:58 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-common master: Remove ceph-specific logic from update/upgrade prepare workflow https://review.openstack.org/585368 | 13:58 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Ceph update and upgrade in config-download era https://review.openstack.org/583321 | 13:58 |
| mwhahaha | #startmeeting tripleo | 14:00 |
| mwhahaha | #topic agenda | 14:00 |
| mwhahaha | * Review past action items | 14:00 |
| mwhahaha | * One off agenda items | 14:00 |
| mwhahaha | * Squad status | 14:00 |
| mwhahaha | * Bugs & Blueprints | 14:00 |
| mwhahaha | * Projects releases or stable backports | 14:00 |
| openstack | Meeting started Tue Jul 24 14:00:05 2018 UTC and is due to finish in 60 minutes. The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| mwhahaha | * Specs | 14:00 |
| mwhahaha | * open discussion | 14:00 |
| mwhahaha | Anyone can use the #link, #action and #info commands, not just the moderatorǃ | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| mwhahaha | Hi everyone! who is around today? | 14:00 |
| *** openstack changes topic to " (Meeting topic: tripleo)" | 14:00 | |
| openstack | The meeting name has been set to 'tripleo' | 14:00 |
| Tengu | «o/ | 14:00 |
| *** openstack changes topic to "agenda (Meeting topic: tripleo)" | 14:00 | |
| *** pradk has joined #tripleo | 14:00 | |
| jfrancoa | \o/ | 14:00 |
| jrist | o/ | 14:00 |
| ksambor | o/ | 14:00 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: undercloud: revert to using the iscsi deploy interface by default https://review.openstack.org/585370 | 14:00 |
| dtantsur | shardy: ^^^ | 14:00 |
| ccamacho1 | hi! | 14:00 |
| *** eck` is now known as eck`gone | 14:00 | |
| rasca | o/ | 14:00 |
| fultonj | o/ | 14:00 |
| slagle | hi | 14:01 |
| marios | o/ | 14:01 |
| chem | o/ | 14:01 |
| panda | o/ | 14:01 |
| owalsh | o/ | 14:01 |
| shardy | o/ | 14:01 |
| jbadiapa | hi | 14:02 |
| dpeacock | o/ | 14:02 |
| myoung | o/ | 14:02 |
| mwhahaha | lets begin | 14:03 |
| mwhahaha | #topic review past action items | 14:03 |
| *** openstack changes topic to "review past action items (Meeting topic: tripleo)" | 14:03 | |
| mwhahaha | Tengu to open a spec + BP for validation framework - DONE | 14:03 |
| mwhahaha | #link https://review.openstack.org/#/c/583475/ | 14:03 |
| Tengu | now things can start ;). | 14:03 |
| mwhahaha | team to help on reviewing https://review.openstack.org/#/q/status:open+topic:alternate_plans | 14:03 |
| mwhahaha | shardy: looks like you -wf that one, are we skipping that for rocky? | 14:03 |
| *** ykarel is now known as ykarel|away | 14:04 | |
| *** eck`gone is now known as eck` | 14:05 | |
| mwhahaha | we'll have to follow up on that later | 14:05 |
| mwhahaha | move https://review.openstack.org/#/c/566448/ to Stein (ping ayoung) | 14:05 |
| mwhahaha | not sure if that was done, I'll follow up with EmilienM | 14:05 |
| mwhahaha | investigate if https://review.openstack.org/#/c/451584/ is for rocky or stein at this point | 14:05 |
| mwhahaha | so the patches for -^ were in merge conflict when i checked yesterday and commented that it'll probably need to be stein | 14:06 |
| *** hamzy_ has quit IRC | 14:07 | |
| mwhahaha | move https://review.openstack.org/#/c/567579/ to stein - NOT DONE | 14:08 |
| shardy | mwhahaha: There are some upgrade issues I'd not anticpated so we may have to defer it | 14:08 |
| mwhahaha | shardy: ok thanks | 14:08 |
| shardy | mwhahaha: I'm planning to continue working on it - the issue is we combine user and t-h-t provided things in the plan-environment.yaml | 14:08 |
| shardy | but Stein may be safer at this point, given we can still deploy e.g OpenShift etc with the current t-h-t | 14:09 |
| mwhahaha | yea that makes more sense given the time left | 14:09 |
| mwhahaha | EmilienM investigates if standalone needs masquerade iptables rules - DONE | 14:10 |
| mwhahaha | #link https://review.openstack.org/#/c/583266/ | 14:10 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 14:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 14:10 |
| mwhahaha | #topic one off agenda items | 14:10 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-meeting-items | 14:10 |
| *** openstack changes topic to "one off agenda items (Meeting topic: tripleo)" | 14:10 | |
| mwhahaha | (myoung) CI Community Meeting starts immediately upon this meeting closing @ https://bluejeans.com/7050859455. All are welcome, ask/discuss anything. https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:10 |
| myoung | ^^ panda has something he would like to discuss afaik | 14:11 |
| panda | yep, the next item is a proposal for the community meeting | 14:11 |
| mwhahaha | (gcerami) We are trying to design the move of OVB jobs to openstack infra. We need some feedback and help for the initial ideas. We can discuss more on the community meeting. | 14:11 |
| *** ykarel|away has quit IRC | 14:12 | |
| panda | moving OVB jobs to openstack-infra, I wanted to start the discussion during PTG, but we are accelerating on that, we have some proposal, and I'd like to involve people to decide fi they are doable | 14:12 |
| *** psahoo has quit IRC | 14:12 | |
| *** agurenko has quit IRC | 14:12 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: undercloud: revert to using the iscsi deploy interface by default https://review.openstack.org/585370 | 14:12 |
| panda | some proposal will ionvolve injecting changes during overcloud deploument | 14:13 |
| panda | other changes after undercloud installation | 14:13 |
| *** kopecmartin has quit IRC | 14:14 | |
| panda | so, I have some question about the workflow | 14:14 |
| panda | if anyone is interested in this discussion please join the community meeting | 14:14 |
| mwhahaha | sounds good | 14:14 |
| mwhahaha | any other items? | 14:15 |
| *** mdnadeem has quit IRC | 14:15 | |
| mwhahaha | sounds like nope, moving on to status | 14:16 |
| mwhahaha | #topic Squad status | 14:16 |
| mwhahaha | ci | 14:16 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:16 |
| mwhahaha | upgrade | 14:16 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status | 14:16 |
| mwhahaha | containers | 14:16 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-containers-squad-status | 14:16 |
| *** openstack changes topic to "Squad status (Meeting topic: tripleo)" | 14:16 | |
| mwhahaha | config-download | 14:16 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-config-download-squad-status | 14:16 |
| mwhahaha | integration | 14:16 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-integration-squad-status | 14:16 |
| mwhahaha | ui/cli | 14:16 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status | 14:16 |
| mwhahaha | validations | 14:16 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-validations-squad-status | 14:16 |
| mwhahaha | networking | 14:16 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-networking-squad-status | 14:16 |
| mwhahaha | workflows | 14:16 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-workflows-squad-status | 14:16 |
| mwhahaha | security | 14:16 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-security-squad | 14:16 |
| *** Haresh has quit IRC | 14:16 | |
| mwhahaha | anything specific folks would like to highlight? | 14:17 |
| Tengu | I would love a few reviews on https://review.openstack.org/#/c/584314/ (validations) | 14:17 |
| jrist | config download reviews and validations reviews needed | 14:17 |
| jrist | ^ | 14:17 |
| *** dtrainor has quit IRC | 14:17 | |
| mwhahaha | jrist: is there a specific topic we can just to review them? | 14:17 |
| *** mjturek has joined #tripleo | 14:17 | |
| jrist | yes! | 14:17 |
| Tengu | main issue is: validation team is kind of decimeted. | 14:17 |
| jrist | https://review.openstack.org/#/q/topic:bp/config-download-ui+(status:open+OR+status:merged) | 14:18 |
| mwhahaha | dat javascript | 14:18 |
| ccamacho1 | Tengo ack Ill add them to my queue | 14:18 |
| ccamacho1 | Tengu** | 14:18 |
| Tengu | ccamacho1: thank you! | 14:18 |
| *** dtrainor has joined #tripleo | 14:19 | |
| mwhahaha | #action team to review outstanding UI changes for config download - https://review.openstack.org/#/q/topic:bp/config-download-ui+(status:open+OR+status:merged) | 14:19 |
| Tengu | and of course, reviews and feedback are needed on https://review.openstack.org/#/c/583475/ (validation framework spec) | 14:19 |
| mwhahaha | sounds good | 14:19 |
| mwhahaha | anything else? | 14:20 |
| * dpeacock chucks https://review.openstack.org/#/c/577397/ into the hat in the hope that someone from validations can +2 | 14:20 | |
| dpeacock | would be great to see that merge | 14:20 |
| Tengu | akrivoka: -^^ :) | 14:20 |
| jrist | mwhahaha: yes a few more, let me find them | 14:21 |
| jrist | https://review.openstack.org/581644 | 14:21 |
| jrist | https://review.openstack.org/#/q/status:open+project:openstack/tripleo-validations+branch:master+topic:openshift-hw-reqs | 14:22 |
| *** wolverineav has quit IRC | 14:23 | |
| *** wolverineav has joined #tripleo | 14:23 | |
| mwhahaha | ok so it sounds like we need some more eyes on validations bits | 14:23 |
| mwhahaha | anyway moving on | 14:23 |
| mwhahaha | #topic bugs & blueprints | 14:24 |
| mwhahaha | #link https://launchpad.net/tripleo/+milestone/rocky-3 | 14:24 |
| mwhahaha | For Rocky we currently have 49 (-5) blueprints and about 740 (+1) open Launchpad bugs. 735 rocky-3, 5 stein-1. 102 (+0) open Storyboard bugs. | 14:24 |
| mwhahaha | #link https://storyboard.openstack.org/#!/project_group/76 | 14:24 |
| *** openstack changes topic to "bugs & blueprints (Meeting topic: tripleo)" | 14:24 | |
| mwhahaha | So Rocky M3 is this week. | 14:24 |
| mwhahaha | if you're blueprint isn't done and doesn't have an FFE we'll be moving them out this week | 14:24 |
| mwhahaha | I sent an email about the outstanding blueprints a few weeks ago | 14:24 |
| mwhahaha | #link http://lists.openstack.org/pipermail/openstack-dev/2018-July/132140.html | 14:25 |
| mwhahaha | so please revisit those blueprints | 14:25 |
| *** bdodd has joined #tripleo | 14:25 | |
| mwhahaha | any other bugs that folks want to highlight? | 14:25 |
| ccamacho1 | Tengo I also added some work items to the blueprint | 14:28 |
| ccamacho1 | there we can agree on the CLI commands and options | 14:28 |
| Tengu | ccamacho1: yep, saw that! | 14:28 |
| mwhahaha | k moving on | 14:28 |
| mwhahaha | #topic projects releases or stable backports | 14:28 |
| *** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)" | 14:28 | |
| mwhahaha | so as I mentioned Rocky M3 is this week | 14:28 |
| Tengu | ccamacho1: shall we discuss that tomorrow? what's your TZ? seems pretty close to CET right? | 14:28 |
| mwhahaha | EmilienM will be posting the patches on Thursday | 14:28 |
| mwhahaha | so anything not landed before that will not be in M3 | 14:29 |
| ccamacho1 | CET indeed, we can sync tomorrow np | 14:29 |
| mwhahaha | EmilienM informed me that he will be doing a stable release next week | 14:29 |
| mwhahaha | Any questions? | 14:29 |
| *** shreshtha has joined #tripleo | 14:29 | |
| *** mcornea has quit IRC | 14:30 | |
| *** links has quit IRC | 14:31 | |
| mwhahaha | sound like nope | 14:31 |
| mwhahaha | #topic specs | 14:31 |
| mwhahaha | #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open | 14:31 |
| *** openstack changes topic to "specs (Meeting topic: tripleo)" | 14:31 | |
| mwhahaha | I previously mentioned some specs that need review | 14:31 |
| mwhahaha | please review them and take some time to make sure any open specs are properly targeted for Stein at this point | 14:31 |
| mwhahaha | #topic open discussion | 14:32 |
| *** openstack changes topic to "open discussion (Meeting topic: tripleo)" | 14:32 | |
| mwhahaha | anything else? | 14:32 |
| mwhahaha | rasca: i saw you were adding something to the agenda, did you have something you wanted to chat about? | 14:32 |
| rasca | mwhahaha, no sorry, my mistake, wrong etherpad | 14:32 |
| mwhahaha | rasca: got it | 14:32 |
| mwhahaha | anyone have anything else? | 14:32 |
| rasca | mwhahaha, thanks for caring | 14:32 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-common stable/pike: Use ansible_queue_name only for upgrade/update https://review.openstack.org/585380 | 14:34 |
| mwhahaha | sounds like nope | 14:35 |
| mwhahaha | myoung, panda i'd like to join the chat about the ovb stuff in the upstream but i'm currently in another meeting. can you hold off on that discussion in the community meeting until i can join? | 14:35 |
| myoung | mwhahaha: ack, and sure. we'll cover rasca's quesrtion first | 14:35 |
| mwhahaha | sounds good | 14:35 |
| mwhahaha | thanks everyone | 14:35 |
| mwhahaha | #endmeeting | 14:35 |
| *** openstack changes topic to "Welcome to Rocky. CI status: GREEN, OVB RED due to nodepool nodefailure, https://trello.com/c/hkvfxAdX | https://docs.openstack.org/tripleo-docs/latest" | 14:35 | |
| openstack | Meeting ended Tue Jul 24 14:35:54 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:35 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-07-24-14.00.html | 14:35 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-07-24-14.00.txt | 14:35 |
| panda | mwhahaha: ok, also I'm on the same meeting | 14:36 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-07-24-14.00.log.html | 14:36 |
| Tengu | so. have to leave """early""". see you tomorrow! | 14:36 |
| myoung | CI Community Meeting starts now @ https://bluejeans.com/7050859455. All are welcome, ask/discuss anything. https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:36 |
| Tengu | mwhahaha: thanks for your review on the vrrp haproxy thingy :) | 14:36 |
| dpeacock | Tengu: later tate | 14:37 |
| dpeacock | +r ;-) | 14:37 |
| rbrady | shardy, jtomasek: I'm interested in continuing the discussion of the CLI / UI features/ux and how that affects the design of the API | 14:37 |
| myoung | panda: you incoming? | 14:37 |
| rbrady | shardy, jtomasek: I'd like to advance the conversation from https://review.openstack.org/#/c/581060/ | 14:38 |
| shardy | rbrady: Hi! Sure happy to discuss | 14:40 |
| openstackgerrit | Jiri Stransky proposed openstack/python-tripleoclient stable/pike: Use only update/upgrade ansible queue instead https://review.openstack.org/585386 | 14:40 |
| shardy | rbrady: I'm facing somewhat similar issues wrt https://review.openstack.org/#/c/574753/ so I think we have a few related problems with the plan-environment API atm | 14:40 |
| *** mcornea has joined #tripleo | 14:40 | |
| rbrady | shardy: I'm wondering if this situation is highlighting that there is a lack of an advocated for the CLI upstream. | 14:41 |
| shardy | the issue with your patch is we have potentially two places where environment ordering comes from, the issue with mine is we have two places where the environment list/order can be defined (the user provided list and the default list in plan-environment) | 14:41 |
| rbrady | shardy: s/advocated/advocate/ | 14:41 |
| shardy | rbrady: yeah - I think we've got a lack of progress aligning the CLI and UI interfaces, e.g I started https://review.openstack.org/#/c/448209/ aiming to help but it stalled | 14:42 |
| shardy | rbrady: I think we've made good progress moving the CLI to use the various workflows, but the environment handling is still a tricky area that differs from the UI | 14:43 |
| rbrady | shardy: I think the alignment / convergence of the CLI and UI needs a more formal approach | 14:43 |
| shardy | rbrady: open to ideas, I was saying to d0ugal this should probably be a PTG topic so we can work out the next steps | 14:43 |
| shardy | the plan management stuff in tripleoclient has been FIXME for a long time now | 14:44 |
| rbrady | shardy: because it's hard to do API design with competeing UXs | 14:44 |
| shardy | but we've not collectively had the cycles to tackle it | 14:44 |
| shardy | rbrady: understood - FWIW I'm not sure they're competing, but we do have some divergent requirements to reconcile | 14:44 |
| shardy | mostly due to $history | 14:44 |
| rbrady | shardy: is this something that everyone should be involved in or is this something that should be driven by the UI/CLI Squad? | 14:44 |
| d0ugal | everyone | 14:45 |
| shardy | ideally we'd align the CLI and UI to use the exact same workflow APIs, but as we know that's hard without breaking interfaces | 14:45 |
| *** janki has joined #tripleo | 14:45 | |
| * jtomasek reads back | 14:45 | |
| shardy | rbrady: I think it's something everyone can probably help with, but the bulk of the work is needed in tripleoclient and the workflows AFAIK | 14:45 |
| shardy | the issue is mostly that we've not had anyone assigned on it as a priority task | 14:46 |
| shardy | which we can probably solve if we can agree on the work required | 14:46 |
| shardy | I think there are two parts | 14:46 |
| shardy | 1. Pass a list of environments to the deploy workflow, stop merging in tripleoclient (I've been discussing this with ramishra recently, as it's related to some heat work he's done and my stalled patch above) | 14:47 |
| shardy | 2. Figure out how to decouple the t-h-t and user provided parts of plan-environment, so we can reasonably update the plan without purging everything | 14:48 |
| shardy | but for (1) we'd still need the list order to be respected, so either we'd need to use the implicit -e order, or have the option to use the capabilities-map | 14:48 |
| shardy | the main issue is the CLI interface has an implicit order which we've long respected, but the UI obviously doesn't | 14:49 |
| *** ksambor has left #tripleo | 14:49 | |
| jtomasek | rbrady, shardy +1, I've already put the topic of UI / CLI alignment to the PTG etherpad | 14:49 |
| *** rpioso|afk is now known as rpioso | 14:49 | |
| shardy | jtomasek: thanks - IIRC we had a session on it in Atlanta but it's probably time to revisit | 14:49 |
| jtomasek | shardy: my point in rbrady's patches was that for backport it would be less work to just backport what rbrady already merged as CLI currently does not use that mistral action | 14:50 |
| *** hamzy_ has joined #tripleo | 14:50 | |
| jrist | lol | 14:50 |
| *** artom has joined #tripleo | 14:50 | |
| jrist | woops wrong chan | 14:51 |
| jrist | not that funny | 14:51 |
| jtomasek | then for next cycle we could update the UpdateCapabilities action to let user specify an ordered list of environments or optionally ask that action to do the per capabilties-map ordering | 14:51 |
| shardy | jtomasek: sure, but it only works because we do the (bad) client side merging, right? | 14:51 |
| jtomasek | IMHO nothing would get broken that way | 14:51 |
| shardy | so it'd be cool to not create yet another barrier to fixing that | 14:51 |
| shardy | if we can just add a boolean to make it optional I think both cases can work? | 14:51 |
| jtomasek | shardy: it would still not work for CLI | 14:52 |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-docs master: Add comments about how to adapt services for FFU https://review.openstack.org/585395 | 14:52 |
| jtomasek | shardy: if it was to use that action | 14:52 |
| *** hamzy_ has quit IRC | 14:52 | |
| *** nenad has quit IRC | 14:52 | |
| jtomasek | shardy: that action currently takes an object where keys are the environments, rather than ordered list of environments | 14:52 |
| *** hamzy_ has joined #tripleo | 14:52 | |
| shardy | jtomasek: so if the CLI specifies a list of environments via plan-environment.yaml (and doesn't merge a single plan-environment) the deploy workflow won't re-order them? | 14:53 |
| jtomasek | shardy: so to convert CLI to use that action, a rework of that action is required anyway | 14:53 |
| *** ksambor has joined #tripleo | 14:53 | |
| rbrady | jtomasek, shardy: would it be helpful to map out what happens in the CLI and web GUI, with a table of what workflows are called and what's done in each UI? | 14:53 |
| *** ksambor has left #tripleo | 14:53 | |
| shardy | jtomasek: ah, Ok so this isn't actually called in the default deploy workflow? | 14:53 |
| jtomasek | shardy: it will because afaik, CLI puts that list directly into swift plan-environment.yaml | 14:53 |
| shardy | rbrady: yes I think that would be very helpful :) | 14:53 |
| shardy | jtomasek: Ok, I think I missed that | 14:53 |
| jtomasek | shardy: the general goal for next cycle would be to make both clients use UpdateCapabilities action (or workflow) to update that part of plan-environment.yaml | 14:54 |
| shardy | if that's the case I won't block the patch but it'd be great to add this to the list of things to discuss at the PTG | 14:54 |
| jtomasek | shardy: absolutely, +1 | 14:54 |
| rbrady | shardy, jtomasek: so maybe for the PTG we can have some sort of diagram of what's happening now and what to focus on for convergence. So for the current patch/backport issue - what can we do to get consensus? It's blocking another workflow patch | 14:55 |
| jtomasek | rbrady: sounds good, I can put together the map of stuff for GUI if you want | 14:55 |
| *** udesale has joined #tripleo | 14:56 | |
| rbrady | jtomasek: thanks! That would be great | 14:56 |
| jtomasek | shardy: regarding the multiple plans support, how about keeping the workflows as it is now (all done in one ceate_plan workflow) and GUI would allow user to (re) set plan type only if there is no deployment | 14:58 |
| shardy | rbrady: about to join a meeting, but I'll revisit the patch after to help unblock the series, thanks for the discussion, happy to help with the CLI diagrams etc for the PTG | 14:58 |
| *** saneax has quit IRC | 14:58 | |
| shardy | jtomasek: yeah but that still doesn't solve the problem on upgrade, because the selected plan-environment is deleted | 14:59 |
| jtomasek | shardy: I have updated the plan creation in GUI and I am bout to integrate the plan selection workflow | 14:59 |
| jtomasek | shardy: ah, right | 14:59 |
| shardy | jtomasek: I've got some ideas on how to fix that but been fighting quickstart all day, lets sync later or tomorrow about the plan selection | 14:59 |
| rbrady | shardy: thanks | 14:59 |
| jtomasek | shardy: sounds good, thanks | 14:59 |
| shardy | it may be we need an interim solution for Rocky but need to discuss the options first | 14:59 |
| *** cshastri has quit IRC | 15:03 | |
| *** lvdombrkr has joined #tripleo | 15:03 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Fix tempest related files permissions for tempest container https://review.openstack.org/584771 | 15:03 |
| *** jcoufal has joined #tripleo | 15:04 | |
| *** dtrainor has quit IRC | 15:04 | |
| *** dtrainor has joined #tripleo | 15:05 | |
| *** rcernin_ has joined #tripleo | 15:05 | |
| *** artom_ has joined #tripleo | 15:08 | |
| *** cylopez has joined #tripleo | 15:09 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 15:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 15:10 |
| *** artom has quit IRC | 15:10 | |
| *** ccamacho1 has quit IRC | 15:18 | |
| *** udesale has quit IRC | 15:18 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-common master: Remove ceph-specific logic from update/upgrade prepare workflow https://review.openstack.org/585368 | 15:19 |
| *** maufart__ has quit IRC | 15:21 | |
| *** ffiore has quit IRC | 15:29 | |
| *** rcernin_ has quit IRC | 15:30 | |
| *** pcaruana has quit IRC | 15:32 | |
| *** msufiyan|lunch has joined #tripleo | 15:34 | |
| *** medberry has quit IRC | 15:35 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Fix tempest related files permissions for tempest container https://review.openstack.org/584771 | 15:35 |
| *** cylopez has left #tripleo | 15:37 | |
| *** leanderthal has quit IRC | 15:38 | |
| *** rlandy is now known as rlandy|brb | 15:42 | |
| EmilienM | mwhahaha: any issues with gate today? | 15:44 |
| EmilienM | http://logs.openstack.org/28/585328/3/check/tripleo-ci-centos-7-undercloud-containers/02b6244/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-07-24_13_08_56 | 15:44 |
| *** lvdombrkr has quit IRC | 15:44 | |
| EmilienM | oh nevermind | 15:44 |
| EmilienM | it's my fault in the patch | 15:45 |
| mwhahaha | yea i haven't seen seen anything big | 15:45 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Add upgrade_tasks for HAproxy https://review.openstack.org/585328 | 15:45 |
| EmilienM | jaosorior: you around? | 15:46 |
| *** wilken[m] has quit IRC | 15:47 | |
| *** itlinux has joined #tripleo | 15:49 | |
| *** yprokule has quit IRC | 15:53 | |
| pabelanger | EmilienM: http://logs.openstack.org/20/583320/3/gate/tripleo-ci-centos-7-undercloud-containers/9fc3e6a/job-output.txt.gz#_2018-07-23_22_00_17_045773 was from last night, think you could use reverse proxy cache here to avoid network outage from rdoproject | 15:54 |
| *** avivgt has quit IRC | 15:54 | |
| *** jfrancoa has quit IRC | 15:58 | |
| *** jpich has quit IRC | 16:01 | |
| *** holser_ has quit IRC | 16:02 | |
| *** agopi|brb has quit IRC | 16:03 | |
| *** agopi|brb has joined #tripleo | 16:03 | |
| EmilienM | pabelanger: will look after lunch | 16:05 |
| *** pradk has quit IRC | 16:09 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 16:10 |
| *** dparkes has quit IRC | 16:13 | |
| *** pradk has joined #tripleo | 16:15 | |
| *** trown is now known as trown|lunch | 16:16 | |
| *** dsneddon has joined #tripleo | 16:17 | |
| *** panda is now known as panda|off | 16:18 | |
| *** pfo has quit IRC | 16:19 | |
| *** zoli is now known as zoli|gone | 16:19 | |
| *** zoli|gone is now known as zoli | 16:19 | |
| *** lucasagomes is now known as lucas-afk | 16:24 | |
| *** ccamacho has joined #tripleo | 16:24 | |
| *** ccamacho has quit IRC | 16:25 | |
| *** ccamacho has joined #tripleo | 16:25 | |
| *** jaganathan has quit IRC | 16:27 | |
| *** rlandy|brb is now known as rlandy | 16:28 | |
| gfidente | sshnaidm|ruck tosky so what are we going to do with https://review.openstack.org/#/c/583587/ ? | 16:33 |
| gfidente | I think it enables some tempest tests | 16:33 |
| tosky | not so much | 16:34 |
| tosky | it just hides the problem vs a real deployment | 16:34 |
| *** dparkes has joined #tripleo | 16:35 | |
| gfidente | tosky it allows running the tests in ci | 16:36 |
| gfidente | but I wanted to see the tests passing, can we make something depend on it to see if they actually pass? | 16:37 |
| tosky | exactly my point (hiding the problem) | 16:37 |
| *** sshnaidm|ruck is now known as sshnaidm|bbl | 16:37 | |
| mwhahaha | sshnaidm|ruck, chkumar|rover: what's the state of ovb? I saw some mirror issues in the latest failures from just a bit ago https://logs.rdoproject.org/13/582913/3/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/75450af/job-output.txt.gz#_2018-07-24_15_56_48_788637 | 16:38 |
| tosky | gfidente: but I don't have +2 on that repository, so it's not up to me | 16:38 |
| gfidente | tosky but the problem it is hiding needs fixing in tempest | 16:39 |
| tosky | I'm just 100% sure that if the workaround is implemented, the real issue will just continue to stay around | 16:39 |
| *** dtantsur is now known as dtantsur|afk | 16:39 | |
| gfidente | tosky or what issue are you thinking about? | 16:39 |
| tosky | fixing the problem in tempest | 16:39 |
| tosky | is there at least a bug for it? | 16:39 |
| gfidente | ah I am all for fixing it in tempest indeed | 16:39 |
| tosky | anyway: neither a core there, nor I'm not going to vote on that | 16:39 |
| gfidente | I think use of skip_path is wrong | 16:39 |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-quickstart-extras master: Ensure pip command is installed before using it https://review.openstack.org/585477 | 16:40 |
| gfidente | tosky filing bug | 16:40 |
| tosky | thanks | 16:40 |
| chkumar|rover | mwhahaha: networking issue in rdo cloud ovs | 16:40 |
| gfidente | tosky you have a link to the submission where I added the comment about /swift/ prefix? | 16:42 |
| gfidente | I lost it :( | 16:42 |
| tosky | gfidente: uh, was it a review? | 16:43 |
| gfidente | tosky I am looking for a link to the three tests in tempest which use skip_path | 16:43 |
| gfidente | I think I found them | 16:44 |
| tosky | do you mean that there was a document which listed them? Or was it just a search on codesearch.openstack.org? | 16:44 |
| *** medberry has joined #tripleo | 16:44 | |
| *** medberry has quit IRC | 16:44 | |
| *** medberry has joined #tripleo | 16:44 | |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-ha-utils master: Introduce latency test https://review.openstack.org/584987 | 16:45 |
| gfidente | tosky can you help triaging this https://bugs.launchpad.net/tempest/+bug/1783369 ? | 16:46 |
| openstack | Launchpad bug 1783369 in tempest "Some ojectstorage tests use skip_path() and assume swift is deployed on the /" [Undecided,New] | 16:46 |
| *** zbitter has joined #tripleo | 16:46 | |
| *** zaneb has quit IRC | 16:47 | |
| *** zbitter is now known as zaneb | 16:48 | |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-ha-utils master: Introduce latency test https://review.openstack.org/584987 | 16:49 |
| chkumar|rover | gfidente: do you have some logs about above bug? | 16:49 |
| *** rlandy is now known as rlandy|afk | 16:49 | |
| *** artom_ has quit IRC | 16:50 | |
| gfidente | chkumar|rover no but if you deploy the swift endpoint in /swift/ | 16:50 |
| gfidente | it will fail | 16:51 |
| gfidente | chkumar|rover /swift/v3 | 16:51 |
| gfidente | vs /v3 | 16:51 |
| *** artom has joined #tripleo | 16:51 | |
| *** derekh has quit IRC | 16:55 | |
| tosky | gfidente: you may add that the default deployment of RadosGW/swift is etc etc | 16:55 |
| chkumar|rover | gfidente: does this jobs http://zuul.openstack.org/builds.html?job_name=legacy-tempest-dsvm-full-devstack-plugin-ceph does not covers it? | 16:58 |
| *** myoung is now known as myoung|lunch | 16:58 | |
| gfidente | chkumar|rover I am not familiar with that job | 17:00 |
| gfidente | but if it is not deploying rgw to replace swift | 17:00 |
| gfidente | or if it is not running the object storage tests | 17:00 |
| gfidente | then it is not covered no | 17:00 |
| *** agopi|brb has quit IRC | 17:03 | |
| *** suuuper has quit IRC | 17:04 | |
| chkumar|rover | EmilienM: mwhahaha sshnaidm|bbl https://review.openstack.org/#/c/583202/ | 17:05 |
| chkumar|rover | gfidente: here is the link of job cinfiguration http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/tempest-dsvm-full-devstack-plugin-ceph/run.yaml | 17:07 |
| *** itlinux_ has joined #tripleo | 17:07 | |
| *** itlinux has quit IRC | 17:09 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 17:10 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 17:10 |
| *** yamahata has quit IRC | 17:11 | |
| *** gfidente has quit IRC | 17:13 | |
| *** pfo has joined #tripleo | 17:14 | |
| *** holser_ has joined #tripleo | 17:15 | |
| openstackgerrit | jadebustos proposed openstack/tripleo-ha-utils master: added config variable for domain in instance-ha resources https://review.openstack.org/585485 | 17:15 |
| *** slaweq has quit IRC | 17:16 | |
| *** ramishra has quit IRC | 17:17 | |
| *** tesseract has quit IRC | 17:17 | |
| *** msufiyan|lunch has quit IRC | 17:17 | |
| mwhahaha | sdoran: do you know if there's an augeas module for ansible | 17:17 |
| sdoran | mwhahaha: I do not but I'll check. What is augeas? | 17:18 |
| mwhahaha | sdoran: http://augeas.net/ | 17:19 |
| mwhahaha | sdoran: we use in puppet to do some file configuration updates (when not just dealing with ini) cause it handles special config file syntax | 17:19 |
| sdoran | That's what I was just reading. Glad I was on the right track. :) | 17:19 |
| *** pfo has quit IRC | 17:19 | |
| mwhahaha | sdoran: it allows us to not have to template crap (tempaltes are awful) | 17:19 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: fix typos in docs and .gitignore https://review.openstack.org/585487 | 17:20 |
| *** ccamacho has quit IRC | 17:20 | |
| *** pfo has joined #tripleo | 17:21 | |
| openstackgerrit | Rafael Folco proposed openstack-infra/tripleo-ci master: WIP: Replace TAGS with ansible var https://review.openstack.org/584508 | 17:21 |
| sdoran | mwhahaha: https://github.com/paluh/ansible-augeas/blob/master/library/augeas.py <= found this. might be a good place to start | 17:22 |
| mwhahaha | sdoran: /me sighs @ https://twitter.com/laserllama/status/416641643933347840?lang=en (so probably not) | 17:22 |
| mwhahaha | let's just duplicate everything | 17:22 |
| mwhahaha | it's cool | 17:22 |
| sdoran | That's old school. | 17:23 |
| mwhahaha | yea lenses are painful to write, but much of the work has already been done for the major formats | 17:25 |
| mwhahaha | templates are just so bad | 17:25 |
| * mwhahaha was looking at chrony ansible roles | 17:25 | |
| sdoran | So we could use that module or write our own for TripleO. | 17:26 |
| sdoran | Not sure it'd be needed upstream, but no reason not to use it in TripleO. | 17:26 |
| sdoran | If we get something working nicely, we can submit it upstream and see if there's any wider community interest. | 17:27 |
| mwhahaha | i just don't want to be templating everything | 17:27 |
| mwhahaha | i know we use it for libvirt and used to use it for docker | 17:27 |
| mwhahaha | the json lense is handy | 17:27 |
| *** mjturek_ has joined #tripleo | 17:28 | |
| *** Haresh has joined #tripleo | 17:29 | |
| *** mjturek has quit IRC | 17:29 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Collectd QDR connection https://review.openstack.org/571152 | 17:36 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Fix Ansible Using tests as filters is deprecated https://review.openstack.org/581779 | 17:36 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: copy ceph config in manila-share container bundle https://review.openstack.org/584949 | 17:36 |
| *** amoralej is now known as amoralej|off | 17:37 | |
| *** trown|lunch is now known as trown | 17:37 | |
| *** shardy has quit IRC | 17:38 | |
| *** pfo has quit IRC | 17:38 | |
| *** itlinux_ has quit IRC | 17:42 | |
| *** dprince has quit IRC | 17:42 | |
| *** itlinux has joined #tripleo | 17:42 | |
| *** yamahata has joined #tripleo | 17:43 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Update the config for FS021 https://review.openstack.org/583202 | 17:45 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Update the config for FS021 https://review.openstack.org/583202 | 17:45 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Update the config for FS021 https://review.openstack.org/583202 | 17:46 |
| EmilienM | chkumar|rover: I took over ^ | 17:46 |
| *** mjturek_ is now known as mjturek | 17:51 | |
| *** hamzy_ has quit IRC | 17:55 | |
| *** dprince has joined #tripleo | 17:57 | |
| *** jaganathan has joined #tripleo | 17:57 | |
| *** Haresh has quit IRC | 18:01 | |
| *** ykarel has joined #tripleo | 18:02 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 18:10 |
| *** slaweq has joined #tripleo | 18:15 | |
| *** gkadam has quit IRC | 18:17 | |
| EmilienM | jillr: in https://review.openstack.org/585075 and all others, I guess you want to make it work on once, then figure out the rest | 18:19 |
| EmilienM | jillr: I think you're missing zuul configuration | 18:19 |
| EmilienM | no job ran, as we can see | 18:19 |
| EmilienM | so we need to figure out why there is not at least basic lint jobs | 18:19 |
| EmilienM | you can easily add them, please create zuul.d/layout.yaml (or something, see another repo for example) | 18:19 |
| EmilienM | https://github.com/openstack/ansible-role-container-registry/blob/master/zuul.d/layout.yaml for ex. | 18:20 |
| *** toure is now known as toure|biab | 18:22 | |
| *** agopi|brb has joined #tripleo | 18:23 | |
| *** agopi|brb is now known as agopi | 18:27 | |
| *** cdearborn has quit IRC | 18:28 | |
| jillr | EmilienM: ack | 18:28 |
| EmilienM | jillr: I would suggest to have one project first, and do others, so we don't waste CI resources | 18:32 |
| jillr | EmilienM: that was the goal with the keystone role, so I've been holding a lot of work locally and was encouraged to WIP it up to gerrit. | 18:33 |
| *** ykarel is now known as ykarel|away | 18:33 | |
| jillr | so, we should pick one approach or another :) | 18:33 |
| EmilienM | jillr: that's fine to have WIPs, but I wouldn't iterate on them until we have at least one of them actually working | 18:36 |
| EmilienM | (passing CI & merged) | 18:36 |
| *** myoung|lunch is now known as myoung | 18:37 | |
| *** hamzy_ has joined #tripleo | 18:39 | |
| jillr | EmilienM: where would I go to find what tripleo-multinode-container-minimal actually does? grep is failing me here. | 18:42 |
| EmilienM | jillr: we don't / want this job now | 18:42 |
| jillr | EmilienM: but I dont see how it would test for example ansible-role-tripleo-keystone until we write any tht that uses the role | 18:42 |
| EmilienM | as it deploys tripleo-multinode-container-minimal jobs | 18:42 |
| EmilienM | but let me show you where it's defined: | 18:42 |
| EmilienM | https://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/zuul.d/multinode-jobs.yaml#n17 | 18:42 |
| EmilienM | we should just run ansible lint job | 18:42 |
| EmilienM | its name is openstack-tox-linters | 18:43 |
| EmilienM | so replace tripleo-multinode-container-minimal by openstack-tox-linters | 18:43 |
| EmilienM | but we need to check with infra if they want this job defined in project-config or not | 18:43 |
| jillr | I've got ansible-lint.sh in there, what am I missing for it? | 18:43 |
| EmilienM | pabelanger: ^ | 18:43 |
| jillr | ah ok | 18:43 |
| EmilienM | pabelanger: can we define openstack-tox-linters in our repo or it has to be in project-config? | 18:43 |
| EmilienM | I remember Ajaeger had an opinion | 18:44 |
| EmilienM | jillr: https://github.com/openstack-infra/project-config/blob/840f986dbbefb4aed7805faba3413695d0efec2e/zuul.d/projects.yaml#L2375-L2386 | 18:45 |
| EmilienM | you need to do that for all the roles | 18:45 |
| EmilienM | so we're consistent with existing roles in tripleo | 18:45 |
| EmilienM | please | 18:45 |
| jillr | for sure, TIL - thanks | 18:45 |
| openstackgerrit | Tom Barron proposed openstack/tripleo-heat-templates stable/queens: copy ceph config in manila-share container bundle https://review.openstack.org/585512 | 18:48 |
| *** mjturek has quit IRC | 18:53 | |
| *** itlinux has quit IRC | 18:54 | |
| openstackgerrit | Merged openstack/tripleo-specs master: fix tox python3 overrides https://review.openstack.org/581206 | 18:54 |
| *** medberry has quit IRC | 18:56 | |
| *** holser_ has quit IRC | 18:59 | |
| *** itlinux has joined #tripleo | 19:02 | |
| *** rlandy|afk is now known as rlandy | 19:03 | |
| *** itlinux has quit IRC | 19:03 | |
| *** eck` is now known as eck`gone | 19:03 | |
| *** ykarel|away is now known as mdnadeem | 19:03 | |
| *** mdnadeem has quit IRC | 19:03 | |
| *** mdnadeem has joined #tripleo | 19:03 | |
| *** mdnadeem has quit IRC | 19:08 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 19:10 |
| *** artom_ has joined #tripleo | 19:22 | |
| *** artom_ has quit IRC | 19:23 | |
| *** artom_ has joined #tripleo | 19:23 | |
| *** artom has quit IRC | 19:25 | |
| EmilienM | something broke tripleo-ci-centos-7-containerized-undercloud-upgrades | 19:28 |
| EmilienM | tripleo-ci-centos-7-containerized-undercloud-upgrades is not upgrading anymore | 19:28 |
| *** toure|biab is now known as toure | 19:34 | |
| *** rwsu has quit IRC | 19:34 | |
| *** pradk has quit IRC | 19:36 | |
| *** agopi is now known as agopi|brb | 19:42 | |
| EmilienM | panda|off, rfolco https://review.openstack.org/#/c/582384/ broke it ^ | 19:44 |
| EmilienM | mwhahaha: containerized undercloud upgrades job isn't testing upgrades since https://review.openstack.org/#/c/582384/ FYI | 19:44 |
| * mwhahaha sighs | 19:44 | |
| EmilienM | https://bugs.launchpad.net/tripleo/+bug/1783399 | 19:46 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] | 19:46 |
| EmilienM | i'll see if we can fix it quick | 19:46 |
| openstackgerrit | Tom Barron proposed openstack/tripleo-heat-templates stable/pike: [PIKE-only] remove experimental manila docker envs https://review.openstack.org/582968 | 19:53 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Fix TAG logic for all upgrade type jobs https://review.openstack.org/585528 | 19:55 |
| EmilienM | mwhahaha: ^ I think that could do it | 19:55 |
| EmilienM | rfolco, panda|off ^ please review it when you can | 19:55 |
| *** artom_ is now known as artom | 19:56 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs050: enable TLS https://review.openstack.org/585333 | 19:56 |
| *** jcoufal has quit IRC | 19:58 | |
| *** artom_ has joined #tripleo | 19:58 | |
| rfolco | EmilienM, looking | 19:58 |
| *** itlinux has joined #tripleo | 19:58 | |
| *** artom has quit IRC | 20:01 | |
| *** artom_ is now known as artom | 20:01 | |
| openstackgerrit | Merged openstack/instack-undercloud stable/pike: Configure keepalived before rabbitmq https://review.openstack.org/585152 | 20:01 |
| *** agopi|brb is now known as agopi | 20:02 | |
| *** eck`gone is now known as eck` | 20:02 | |
| pabelanger | EmilienM: it has to stay in project-config, but you can just parent to it | 20:04 |
| pabelanger | or write a new one | 20:04 |
| *** pfo has joined #tripleo | 20:04 | |
| EmilienM | pabelanger: k | 20:05 |
| *** mjturek has joined #tripleo | 20:08 | |
| EmilienM | rfolco: we need to be super careful with these vars | 20:10 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 20:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 20:10 |
| *** pradk has joined #tripleo | 20:10 | |
| *** sshnaidm|bbl is now known as sshnaidm|ruck | 20:11 | |
| *** pfo_ has joined #tripleo | 20:12 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Correct vrrp script for haproxy status https://review.openstack.org/583886 | 20:12 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Update/correct vrrp check for haproxy https://review.openstack.org/584754 | 20:12 |
| *** pfo has quit IRC | 20:15 | |
| *** rcernin_ has joined #tripleo | 20:19 | |
| *** jtomasek has quit IRC | 20:20 | |
| *** shreshtha has quit IRC | 20:22 | |
| *** chem has quit IRC | 20:22 | |
| *** jtcressy has joined #tripleo | 20:24 | |
| sdoran | mwhahaha: https://review.openstack.org/#/c/585500/ https://review.openstack.org/#/c/583724/ <= could you please review? | 20:24 |
| EmilienM | sdoran: ack, will do before eod | 20:27 |
| *** bugzy has joined #tripleo | 20:28 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo master: Release Rocky m3: 9.2.0 https://review.openstack.org/585537 | 20:30 |
| *** agopi has left #tripleo | 20:30 | |
| *** agopi has joined #tripleo | 20:30 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-ui master: Release Rocky M3: 9.2.0 https://review.openstack.org/585538 | 20:30 |
| EmilienM | sdoran, jillr : it won't land before we fix zuul layout | 20:30 |
| EmilienM | no job ran | 20:31 |
| *** janki has quit IRC | 20:31 | |
| *** bugzy_ has quit IRC | 20:31 | |
| sdoran | Hmmmm, so I need to define a Zuul job for the role? | 20:31 |
| mwhahaha | EmilienM: so they asked about that when we created the repos and I said we'd probably just do the zuul config in tree | 20:33 |
| EmilienM | sdoran: let me show you, one sec | 20:33 |
| EmilienM | ah ok | 20:33 |
| EmilienM | anyway, let me do it in tree then | 20:33 |
| mwhahaha | EmilienM, sdoran, jillr: so we need to create simple .zuul.yaml | 20:33 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-upgrade master: DNM: just test rdo cloud multinode https://review.openstack.org/585539 | 20:34 |
| *** medberry has joined #tripleo | 20:35 | |
| *** medberry has quit IRC | 20:35 | |
| *** medberry has joined #tripleo | 20:35 | |
| EmilienM | jillr, sdoran: we need this patch first https://review.openstack.org/585542, and please rebase your patches on top of this one | 20:36 |
| EmilienM | also there is no IRC notification, it was missed during the project creation probably? | 20:36 |
| *** paramite has quit IRC | 20:36 | |
| EmilienM | sdoran: FYI https://review.openstack.org/585543 | 20:38 |
| EmilienM | jillr, sdoran : I also rebased all your patches for you, on top of mine. | 20:39 |
| EmilienM | you can check in http://zuul.openstack.org/, the lint job is now running | 20:39 |
| *** d0ugal has quit IRC | 20:44 | |
| *** ansmith has quit IRC | 20:44 | |
| jillr | EmilienM: mwhahaha I dont readily see anywhere to reference a file named .zuul.yaml - is this in addition to zuul.d/layout.yaml? | 20:45 |
| openstackgerrit | Merged openstack/os-net-config master: Stub out check for OVS installed to avoid failing tests https://review.openstack.org/585070 | 20:46 |
| openstackgerrit | Merged openstack/tripleo-common master: Generate additional roles with defined roles file https://review.openstack.org/583165 | 20:46 |
| mwhahaha | jillr: it's a magic config, either .zuul.yaml or zuul.d/layout.yaml works | 20:46 |
| jillr | ahk | 20:47 |
| *** artom has quit IRC | 20:47 | |
| mwhahaha | .zuul.yaml was the original name, i think zuul.d/layout.yal is the prefered file now | 20:47 |
| openstackgerrit | Jill Rouleau proposed openstack/ansible-role-tripleo-keystone master: Add basic jobs in Zuul Layout https://review.openstack.org/585548 | 20:48 |
| EmilienM | jillr: its all the same, yeah | 20:48 |
| EmilienM | mwhahaha: I like having zuul.d better, so we can structure things into a directory | 20:49 |
| EmilienM | and not having one single YAML in .zuul.yaml | 20:49 |
| EmilienM | https://github.com/openstack-infra/tripleo-ci/tree/master/zuul.d | 20:49 |
| EmilienM | like this ^ | 20:49 |
| *** mjturek has quit IRC | 20:49 | |
| *** agopi is now known as agopi|brb | 20:51 | |
| *** morazi has quit IRC | 20:51 | |
| *** mjturek has joined #tripleo | 20:52 | |
| *** hamzy_ has quit IRC | 20:55 | |
| *** lblanchard has quit IRC | 20:57 | |
| *** mjturek has quit IRC | 20:57 | |
| *** trown is now known as trown|outtypewww | 21:02 | |
| *** gfidente has joined #tripleo | 21:05 | |
| *** gfidente has quit IRC | 21:05 | |
| *** gfidente has joined #tripleo | 21:05 | |
| *** mjturek has joined #tripleo | 21:06 | |
| gfidente | dear people-who-cares-about-swift-or-haproxy | 21:09 |
| gfidente | I think this is fine to go https://review.openstack.org/#/c/582991/ | 21:09 |
| *** mjturek has quit IRC | 21:09 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 21:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 21:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 21:10 |
| *** mjturek has joined #tripleo | 21:12 | |
| *** pchavva has quit IRC | 21:13 | |
| *** brault has quit IRC | 21:17 | |
| *** pradk has quit IRC | 21:20 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Include minimal Browbeat playbook in baremetal playbook https://review.openstack.org/581488 | 21:26 |
| *** bugzy_ has joined #tripleo | 21:27 | |
| *** rcernin_ has quit IRC | 21:30 | |
| *** bugzy has quit IRC | 21:30 | |
| *** slaweq has quit IRC | 21:30 | |
| *** jtcressy has quit IRC | 21:31 | |
| *** edmondsw has quit IRC | 21:34 | |
| *** agopi|brb is now known as agopi | 21:35 | |
| *** itlinux has quit IRC | 21:38 | |
| *** ansmith has joined #tripleo | 21:39 | |
| *** brault has joined #tripleo | 21:41 | |
| *** bugzy has joined #tripleo | 21:43 | |
| *** bugzy_ has quit IRC | 21:45 | |
| *** agopi is now known as agopi|brv | 21:48 | |
| *** jtcressy has joined #tripleo | 21:49 | |
| pabelanger | EmilienM: who is best to explain http://logs.openstack.org/90/585190/1/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/bd0409f/job-output.txt.gz#_2018-07-24_21_34_15_134831 | 21:50 |
| pabelanger | and why we need to create mkdir -p /home/zuul/src/git.openstack.org/openstack//tripleo-ci | 21:50 |
| jtcressy | mwhahaha: looks like I failed on workflow step 2 execution, but this time it was only one node that failed. Take a look at the last few lines: https://hastebin.com/nilujosege | 21:52 |
| *** agopi|brv has quit IRC | 21:53 | |
| jtcressy | "Error ENOENT: unrecognized pool 'images'" | 21:53 |
| jtcressy | the node at .7 seems to not be recognizing the storage pools? | 21:53 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Fix parameter name used to create the Manila CephX keyring https://review.openstack.org/585562 | 21:56 |
| *** dprince has quit IRC | 22:02 | |
| *** tvignaud has quit IRC | 22:03 | |
| *** tosky has quit IRC | 22:07 | |
| *** jtcressy has quit IRC | 22:07 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 22:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 22:10 |
| *** tvignaud has joined #tripleo | 22:10 | |
| *** jtcressy has joined #tripleo | 22:10 | |
| *** slaweq has joined #tripleo | 22:11 | |
| *** slaweq has quit IRC | 22:16 | |
| mwhahaha | jtcressy: googling points to possibly a bad configuration https://bugzilla.redhat.com/show_bug.cgi?id=1592508 | 22:20 |
| openstack | bugzilla.redhat.com bug 1592508 in puppet-ceph "Ceph pool customize parameter are not passed to Ceph Ansible" [Medium,Closed: notabug] - Assigned to gfidente | 22:20 |
| mwhahaha | jtcressy: the internal comment indicates CephPool parameter should not be nested within CephAnsibleDisksConfig | 22:20 |
| * mwhahaha also throws things at gfidente for not leaving that comment public :D | 22:21 | |
| *** medberry has quit IRC | 22:23 | |
| *** toure is now known as toure|gone | 22:27 | |
| *** rcernin has joined #tripleo | 22:30 | |
| gfidente | mwhahaha well that's because that was user error | 22:32 |
| gfidente | jtcressy not sure if that is your issue | 22:32 |
| mwhahaha | should still leave that public so folks know why this might happen | 22:32 |
| gfidente | mwhahaha ok done, let me check upstream docs | 22:32 |
| jtcressy | I don't have CephPoolDefaultSize nested within CephAnsibleDisksConfig though.. none of it is nested. | 22:33 |
| gfidente | jtcressy I see it was creating 1024 pgs so this is not same issue | 22:33 |
| jtcressy | Here's my yaml config for ceph https://hastebin.com/vijuzekuto.js | 22:33 |
| gfidente | jtcressy if you paste the full ceph-ansible log we might find the cause | 22:34 |
| gfidente | jtcressy how many osds are you deploying? | 22:35 |
| jtcressy | The full log from /var/log/mistral exceeds hastebin's 400,000 character limit | 22:35 |
| jtcressy | 18 OSD's | 22:35 |
| gfidente | so you can only host 3600 pgs in total | 22:35 |
| jtcressy | 6 drives in each of my 3 ceph nodes | 22:35 |
| gfidente | if you set the default to 1024, considering we create at least 5 pools (backups, vms, images, volumes, metrics) | 22:37 |
| gfidente | you'd need much more than 3600 | 22:37 |
| gfidente | so it is probably failing earlier with the cluster refusing to create some | 22:37 |
| jtcressy | oh, I didnt know it was creating 5 pools | 22:37 |
| *** rlandy has quit IRC | 22:37 | |
| gfidente | jtcressy you have two options | 22:37 |
| gfidente | one is to set a lower default number and use a bigger only for the pools you want to spread more | 22:38 |
| gfidente | not sure if makes any sense having 1024 pgs with 6*3 osds | 22:38 |
| jtcressy | I guess I could set the default pg to 600 | 22:38 |
| jtcressy | when I ran the calculator it assumed one pool | 22:38 |
| gfidente | ceph won't spread over more than 3 osds anyway because it looks for different hosts | 22:39 |
| gfidente | does not replicate across osds on the same host | 22:39 |
| gfidente | so you could set a much lower default | 22:39 |
| gfidente | and keep the value a power of 2 for performances | 22:39 |
| jtcressy | so 256 | 22:39 |
| gfidente | that or the default, 128, will also work | 22:39 |
| gfidente | you can increase the pgsize after the deployment if you need | 22:40 |
| gfidente | or you can use CephPools to override the default for specific pools | 22:40 |
| gfidente | on the new attempt, remember to clean the disks :D | 22:41 |
| jtcressy | I wish there was a "clean" button on ironic.... it'll get HUGEly cumbersome when the bigger production deployment has more than a handful of ceph nodes. | 22:43 |
| gfidente | jtcressy check this stuff https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ceph_config.html | 22:43 |
| mwhahaha | there is | 22:43 |
| mwhahaha | it just takes a long time | 22:43 |
| jtcressy | I might be ok with it taking a long time. | 22:43 |
| mwhahaha | jtcressy: clean_nodes on the undercloud.conf | 22:43 |
| mwhahaha | set to true | 22:44 |
| jtcressy | don't I have to reinstall the undercloud to do that though? | 22:44 |
| gfidente | and note I spotted an error already, it's ANSIBLE_FORKS not DEFAULT_FORKS | 22:44 |
| mwhahaha | no you should be able to update the conf and just rerun the install | 22:44 |
| gfidente | mwhahaha jtcressy you should be able to update the existing config yes, but you can't shrink the pgsize for the pools that have been created already | 22:45 |
| gfidente | mwhahaha so the new default should be small enough to let the missing pools fit within the remaining PGs | 22:45 |
| gfidente | THE CAKE IS A LIE! | 22:45 |
| mwhahaha | but it's a delicious lie | 22:45 |
| gfidente | yeah this is an unfortunate scenario | 22:46 |
| gfidente | to recover from | 22:46 |
| mwhahaha | story of dealing with ceph | 22:46 |
| * mwhahaha ducks | 22:46 | |
| gfidente | well you could delete the pools | 22:46 |
| gfidente | and let ceph-ansible retry | 22:46 |
| gfidente | with the new defaults | 22:47 |
| gfidente | this should actually work | 22:47 |
| gfidente | and we get a piece of cake | 22:47 |
| gfidente | jtcressy and regarding ironic cleaning disks, check https://docs.openstack.org/ironic/latest/admin/cleaning.html | 22:49 |
| jtcressy | Got it, i'm manually cleaning the ceph nodes right now using ironic. | 22:50 |
| mwhahaha | yea i fyou enable clean_nodes on the undercloud, it'll do it for you when you delete a stack | 22:50 |
| mwhahaha | takes a while though | 22:50 |
| jtcressy | I dont need it to clean every node on every deploy, so I dont need clean_nodes=true. I'll just run the baremetal node clean commands every time. | 22:50 |
| *** slaweq has joined #tripleo | 22:50 | |
| gfidente | I can see the candles | 22:50 |
| gfidente | also, if you enable manila/mds or rgw you need even more pools | 22:51 |
| gfidente | and using rgw when deploying ceph is probably a good idea | 22:52 |
| gfidente | pools count jumps to 8 | 22:52 |
| gfidente | environments/ceph-ansible/ceph-rgw.yaml that is | 22:52 |
| jtcressy | gfidente: what variables do I set to enable rgw? and is this ok with 18 OSD's on 3 nodes? CephPoolDefaultSize: 3 | 22:53 |
| gfidente | yes size it set to 3 by default as well, 3 is good enough | 22:53 |
| gfidente | you can raise it for some pools if disks replacement leaves you with <3 for longer period of times | 22:53 |
| *** yrabl has quit IRC | 22:54 | |
| gfidente | the above env file is all is needed to enable rgw to replace swift | 22:54 |
| jtcressy | If I deploy a huge system with a bunch of disks i'll probably want to do a pool size of 5 or something. | 22:54 |
| *** slaweq has quit IRC | 22:54 | |
| gfidente | jtcressy I think key factor is go setting these values also on a per-pool basis | 22:55 |
| gfidente | but ceph won't shrink | 22:55 |
| gfidente | so start safe | 22:55 |
| gfidente | (actually size can be lowered, but not pgs) | 22:55 |
| jtcressy | this is what i'm going to use in my next deployment attempt: https://hastebin.com/omijizujus | 22:56 |
| gfidente | *also on a per-pool basis means specific values for those pools which benefit | 22:56 |
| jtcressy | Will rgw be enabled on its own? | 22:57 |
| gfidente | jtcressy you need -e environments/ceph-ansible/ceph-rgw.yaml | 22:57 |
| gfidente | to enable rgw | 22:57 |
| jtcressy | ok | 22:57 |
| jtcressy | here's my deployment command: https://hastebin.com/okoharifok | 22:58 |
| gfidente | jtcressy with 3 physical nodes you can't have size 5 anyway | 22:58 |
| gfidente | and cluster stays HEALTH_WARN | 22:58 |
| jtcressy | I know, thats why I said in a larger deployment i'd want to do 5. | 22:58 |
| gfidente | jtcressy you miss $THT before the rgw line | 22:58 |
| gfidente | and also, another piece of cake | 22:58 |
| jtcressy | whoops thanks | 22:58 |
| gfidente | the order of the -e is important | 22:59 |
| gfidente | to make sure your customizations in ceph-osd-config.yaml preveal on the other files pass if after the ceph-ansible env files | 22:59 |
| gfidente | *pass it | 22:59 |
| jtcressy | got it. | 23:00 |
| *** mschuppert has quit IRC | 23:03 | |
| *** rfolco has quit IRC | 23:04 | |
| gfidente | jtcressy trying to be serious, the try/repeat dynamics are still a good learning exercise, if you find stuff missing in the docs, push fixes to tripleo-docs | 23:04 |
| *** rfolco has joined #tripleo | 23:05 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Move localloop to exec_sudo https://review.openstack.org/578616 | 23:05 |
| openstackgerrit | Merged openstack/diskimage-builder master: Call kpartx remove in umount, not cleanup https://review.openstack.org/578657 | 23:05 |
| *** rh-jelabarre has quit IRC | 23:07 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1782267 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783055 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 23:10 |
| openstack | Launchpad bug 1783055 in tripleo "[master]multinode periodic promotion jobs are failing where Tempest is run in container" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 23:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 23:10 |
| *** slaweq has joined #tripleo | 23:11 | |
| *** rh-jelabarre has joined #tripleo | 23:13 | |
| *** pmannidi has joined #tripleo | 23:14 | |
| *** slaweq has quit IRC | 23:16 | |
| *** pfo_ has quit IRC | 23:18 | |
| *** pfo_ has joined #tripleo | 23:19 | |
| *** agopi has joined #tripleo | 23:25 | |
| *** wolverineav has quit IRC | 23:26 | |
| *** wolverineav has joined #tripleo | 23:27 | |
| *** alee has quit IRC | 23:31 | |
| *** wolverineav has quit IRC | 23:33 | |
| *** mcornea has quit IRC | 23:35 | |
| *** gfidente has quit IRC | 23:37 | |
| openstackgerrit | Merged openstack/tripleo-upgrade master: cont/uc: remove --use-heat https://review.openstack.org/585330 | 23:39 |
| *** rpioso is now known as rpioso|afk | 23:51 | |
| *** mjturek has quit IRC | 23:54 | |
| *** yolanda_ has joined #tripleo | 23:56 | |
| *** pfo_ has quit IRC | 23:58 | |
| *** yolanda has quit IRC | 23:58 | |
| *** wolverineav has joined #tripleo | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!