| *** dtantsur|afk has quit IRC | 00:00 | |
| weshay | tonyb, I only see it under check so far | 00:00 |
|---|---|---|
| weshay | and that it's non-voting in-fact | 00:01 |
| *** dtantsur has joined #tripleo | 00:01 | |
| tonyb | http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n103 | 00:02 |
| tonyb | which puts it in gate on master | 00:02 |
| tonyb | IIUC | 00:02 |
| *** morazi has quit IRC | 00:03 | |
| weshay | tonyb, ya.. that looks like it | 00:03 |
| weshay | tonyb, thank you | 00:04 |
| tonyb | weshay: np gald I can help instead of asking questions | 00:04 |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 00:10 |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 00:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 00:10 |
| *** etingof has quit IRC | 00:10 | |
| EmilienM | mwhahaha, weshay: I don't get what's wrong with https://review.openstack.org/#/c/513163/4 tbh | 00:18 |
| EmilienM | job is failing? | 00:18 |
| EmilienM | it's supposed to work, no? | 00:18 |
| weshay | failing in da gates | 00:21 |
| *** bobh has quit IRC | 00:21 | |
| weshay | https://review.openstack.org/#/c/517023/ | 00:21 |
| weshay | https://review.openstack.org/#/c/514576/ | 00:22 |
| EmilienM | weshay: ok so revert? | 00:22 |
| *** oanson has quit IRC | 00:22 | |
| *** oanson has joined #tripleo | 00:24 | |
| *** zshi has joined #tripleo | 00:25 | |
| weshay | EmilienM, don't know | 00:25 |
| *** bobh has joined #tripleo | 00:25 | |
| EmilienM | weshay: if nobody is able to maintain this job in stable state, let's revert | 00:26 |
| EmilienM | like if nobody shows up to fix it, ok disable it | 00:26 |
| EmilienM | if anyone complains, go fix it | 00:26 |
| EmilienM | how does it sound? | 00:26 |
| EmilienM | I mean, we can't maintain everything. | 00:26 |
| weshay | EmilienM, ack.. it's going away soon anyway | 00:26 |
| EmilienM | no worries I'll take care of that. | 00:26 |
| EmilienM | weshay: wait | 00:26 |
| EmilienM | where is the failure | 00:26 |
| EmilienM | I'll investigate 5 Min and revert otherwise | 00:26 |
| weshay | http://logs.openstack.org/23/517023/2/gate/legacy-tripleo-ci-centos-7-undercloud-containers/d8c10e0/job-output.txt.gz#_2017-11-02_12_45_24_371737 | 00:27 |
| EmilienM | Slower: you around by any chance? ^ | 00:27 |
| weshay | Slower, get off your sled and get over here | 00:28 |
| EmilienM | lol | 00:28 |
| *** bobh has quit IRC | 00:29 | |
| mwhahaha | EmilienM: undercloud-containers nv in check, voting in gate | 00:29 |
| EmilienM | let's fix it? | 00:30 |
| EmilienM | I mean what's the problem | 00:30 |
| EmilienM | I'm busy 5 min and I'll debug before -nv it | 00:30 |
| weshay | it really needs to be voting in check for it to gate | 00:31 |
| mwhahaha | Right it failed which is what lead me to notice this inconsistency | 00:31 |
| mwhahaha | I don't know why it failed but that wasnty concern | 00:31 |
| weshay | what the f.. is wasnty | 00:32 |
| mwhahaha | More of we have a nonvoting check job that's votes in gate | 00:32 |
| mwhahaha | Wasn't my | 00:32 |
| mwhahaha | Damn you auto correct | 00:32 |
| weshay | mwhahaha, soosh, I'm busting his chops | 00:32 |
| mwhahaha | It's 6:30, wasnty is an acceptable form of English :D | 00:33 |
| weshay | heh | 00:33 |
| weshay | I think it's only 4:30 for him | 00:34 |
| *** etingof has joined #tripleo | 00:34 | |
| *** noslzzp has quit IRC | 00:34 | |
| *** noslzzp has joined #tripleo | 00:35 | |
| tonyb | huzzah docker ps isn't empty! it turns out pointhing at the right namespace and prefix is helpful | 00:36 |
| EmilienM | weshay: for who? | 00:36 |
| *** eck`gone is now known as eck` | 00:43 | |
| EmilienM | I'm back and debugging | 00:48 |
| EmilienM | it sounds random | 00:51 |
| EmilienM | it sounds like no container was running http://logs.openstack.org/23/517023/2/gate/legacy-tripleo-ci-centos-7-undercloud-containers/d8c10e0/logs/var/log/extra/docker/docker_allinfo.txt.gz | 00:52 |
| *** brault has joined #tripleo | 00:53 | |
| *** brault has quit IRC | 00:56 | |
| *** brault has joined #tripleo | 00:57 | |
| EmilienM | weshay, mwhahaha: legacy-tripleo-ci-centos-7-undercloud-containers is passing in a fairly good % isn't? | 00:57 |
| EmilienM | check http://cistatus.tripleo.org/ | 00:57 |
| EmilienM | 80% passed yesterday | 00:57 |
| EmilienM | 80% overall | 00:57 |
| *** jcoufal has quit IRC | 00:59 | |
| weshay | EmilienM, no.. ur looking at the wrong thing | 00:59 |
| weshay | 74% today 69% yesterday | 01:00 |
| *** jcoufal has joined #tripleo | 01:00 | |
| *** agopi_ has joined #tripleo | 01:00 | |
| weshay | look at the graphs below the job name after clicking on it | 01:00 |
| EmilienM | 74, not 80, SORRY | 01:00 |
| EmilienM | lol | 01:00 |
| EmilienM | I mean if it's bad, ok let's revert | 01:00 |
| weshay | comparing apples to apples | 01:00 |
| weshay | legacy-tripleo-ci-centos-7-undercloud-oooq | 01:01 |
| *** brault has quit IRC | 01:01 | |
| weshay | is at 95% today, 96.8 yesterday | 01:01 |
| EmilienM | no I checked legacy-tripleo-ci-centos-7-undercloud-containers | 01:01 |
| EmilienM | ok anyway | 01:01 |
| weshay | right.. I'm comparing the two undercloud jobs | 01:01 |
| weshay | old vs. containers | 01:01 |
| weshay | overall old 92.6 vs containers 78% atm | 01:02 |
| weshay | well.. for the last two weeks | 01:02 |
| tonyb | so I have a mysql container and it looks to be running, but the IP that it shoudl be on still seems to be unallocated and heat isn't about to reach it: "Can't connect to MySQL server on '192.168.0.65' ([Errno 113] No route to host)") | 01:03 |
| EmilienM | "Error: /Stage[main]/Tripleo::Profile::Base::Logging::Fluentd/User[fluentd]/ensure: change from absent to present failed: Could not create user fluentd: Execution of '/usr/sbin/useradd -G ceilometer,cinder,glance,gnocchi,heat,keystone,neutron,nova,root -M fluentd' returned 6: useradd: group 'ceilometer' does not exist", | 01:03 |
| EmilienM | mwhahaha: ^ does it ring a bell to you? | 01:03 |
| tonyb | I'm still not sure how to work out what should be handling the IP setup | 01:03 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Temporarily disable fluentd from scenario001-multinode-containers https://review.openstack.org/517497 | 01:04 |
| EmilienM | mwhahaha: nevermind, foun dit | 01:04 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Temporarily disable fluentd from scenario001-multinode-containers https://review.openstack.org/517497 | 01:04 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Add --detailed-exitcodes when running puppet via ansible https://review.openstack.org/517022 | 01:05 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: DNM - Test Puppet exit code in Pike https://review.openstack.org/517440 | 01:05 |
| EmilienM | weshay: what do you want me to do? revert? disable voting? quit? | 01:07 |
| EmilienM | I vote for #3 | 01:07 |
| weshay | #3 | 01:07 |
| weshay | EmilienM, I could send an email to openstack-dev[tripleo] | 01:08 |
| weshay | requesting their attention | 01:08 |
| EmilienM | ok | 01:08 |
| EmilienM | thanks | 01:08 |
| EmilienM | good night, I'm out | 01:08 |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 01:10 |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 01:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 01:10 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common master: Updated from global requirements https://review.openstack.org/511633 | 01:13 |
| *** achadha_ has joined #tripleo | 01:16 | |
| *** achadha__ has joined #tripleo | 01:18 | |
| *** achadha has quit IRC | 01:20 | |
| *** achadha_ has quit IRC | 01:21 | |
| *** noslzzp has quit IRC | 01:21 | |
| *** achadha__ has quit IRC | 01:22 | |
| *** achadha has joined #tripleo | 01:23 | |
| *** achadha has quit IRC | 01:27 | |
| *** noslzzp has joined #tripleo | 01:31 | |
| *** itlinux has joined #tripleo | 01:32 | |
| *** rlandy|afk is now known as rlandy | 01:36 | |
| *** psachin has joined #tripleo | 01:37 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common master: Updated from global requirements https://review.openstack.org/511633 | 01:42 |
| *** jcoufal has quit IRC | 01:44 | |
| *** jcoufal has joined #tripleo | 01:45 | |
| *** oanson has quit IRC | 01:52 | |
| *** oanson has joined #tripleo | 01:52 | |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient master: Add --output-dir https://review.openstack.org/514789 | 01:52 |
| *** rlandy has quit IRC | 01:53 | |
| *** noslzzp has quit IRC | 01:59 | |
| *** jcoufal has quit IRC | 01:59 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart-extras master: config-download: use workflow supported in Queens https://review.openstack.org/516841 | 02:00 |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart master: fs010: use --config-download from tripleoclient https://review.openstack.org/514736 | 02:00 |
| *** yamahata has quit IRC | 02:02 | |
| *** itlinux has quit IRC | 02:03 | |
| *** jcoufal has joined #tripleo | 02:03 | |
| *** moshele has quit IRC | 02:08 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Add workflow for deployment via config-download https://review.openstack.org/512876 | 02:09 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Log Ansible actions to ansible.log https://review.openstack.org/514783 | 02:09 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Add work_dir for some actions https://review.openstack.org/514784 | 02:09 |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 02:10 |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 02:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 02:10 |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart master: fs010: use --config-download from tripleoclient https://review.openstack.org/514736 | 02:11 |
| *** moshele has joined #tripleo | 02:13 | |
| *** jcoufal has quit IRC | 02:16 | |
| *** achadha has joined #tripleo | 02:20 | |
| *** achadha has quit IRC | 02:25 | |
| *** moshele has quit IRC | 02:34 | |
| *** jcoufal has joined #tripleo | 02:36 | |
| *** etingof has quit IRC | 02:36 | |
| *** agopi_ has quit IRC | 02:49 | |
| *** etingof has joined #tripleo | 02:50 | |
| *** ecerquei has joined #tripleo | 02:52 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Update CephPools format in the docker templates to fit ceph-ansible https://review.openstack.org/512274 | 02:53 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: HAProxy: Mount the CA certificate if TLS everywhere is enabled https://review.openstack.org/517172 | 02:53 |
| *** noslzzp has joined #tripleo | 02:54 | |
| *** fragatina has quit IRC | 02:56 | |
| *** fragatina has joined #tripleo | 02:57 | |
| *** fragatin_ has joined #tripleo | 02:59 | |
| *** fragati__ has joined #tripleo | 03:01 | |
| *** fragatina has quit IRC | 03:01 | |
| *** noslzzp has quit IRC | 03:03 | |
| *** noslzzp has joined #tripleo | 03:03 | |
| *** fragatin_ has quit IRC | 03:03 | |
| *** fragati__ has quit IRC | 03:04 | |
| *** noslzzp has quit IRC | 03:04 | |
| *** fragatina has joined #tripleo | 03:05 | |
| *** fragatina has quit IRC | 03:09 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Providing required priviledges to the mounted NFS volume https://review.openstack.org/510846 | 03:09 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Add NetIpMap to hieradata for *ExtraConfig overrides https://review.openstack.org/517264 | 03:09 |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 03:10 |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 03:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 03:10 |
| *** jcoufal has quit IRC | 03:13 | |
| openstackgerrit | Keith Schincke proposed openstack/puppet-tripleo master: Set ACLs on ceph client keyrings https://review.openstack.org/517347 | 03:15 |
| *** ecerquei has quit IRC | 03:24 | |
| *** ramishra has joined #tripleo | 03:24 | |
| *** achadha has joined #tripleo | 03:36 | |
| *** achadha_ has joined #tripleo | 03:37 | |
| *** achadha has quit IRC | 03:41 | |
| *** shreshtha has joined #tripleo | 03:47 | |
| *** shreshtha has quit IRC | 03:47 | |
| *** shreshtha has joined #tripleo | 03:48 | |
| *** udesale has joined #tripleo | 03:48 | |
| *** achadha_ has quit IRC | 03:50 | |
| *** achadha has joined #tripleo | 03:51 | |
| *** achadha has quit IRC | 03:55 | |
| *** fragatina has joined #tripleo | 04:07 | |
| *** anilvenkata has joined #tripleo | 04:09 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 04:10 |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 04:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 04:10 |
| *** anilvenkata has quit IRC | 04:10 | |
| *** anilvenkata has joined #tripleo | 04:10 | |
| *** fragatina has quit IRC | 04:11 | |
| *** ykarel|away is now known as ykarel | 04:29 | |
| *** skramaja has joined #tripleo | 04:39 | |
| *** agopi_ has joined #tripleo | 04:39 | |
| *** pdeore has joined #tripleo | 04:51 | |
| *** achadha has joined #tripleo | 04:58 | |
| *** achadha has quit IRC | 05:03 | |
| *** brault has joined #tripleo | 05:05 | |
| *** janki has joined #tripleo | 05:08 | |
| *** yamahata has joined #tripleo | 05:09 | |
| *** brault has quit IRC | 05:09 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 05:10 |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 05:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 05:10 |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-heat-templates master: Add trunk service plugin to OVN environment files https://review.openstack.org/502542 | 05:14 |
| *** chlong has quit IRC | 05:26 | |
| *** oanson has quit IRC | 05:34 | |
| *** oanson has joined #tripleo | 05:34 | |
| *** cshastri has joined #tripleo | 05:44 | |
| *** ratailor has joined #tripleo | 05:45 | |
| *** jaganathan has joined #tripleo | 05:50 | |
| Tengu | hello there! | 05:54 |
| Tengu | o___O I think I just hit some bug in the CI | 05:56 |
| Tengu | ### undercloud-ram FAILED ### The RAM on the undercloud node is 7978 MB, the minimal recommended value is 16384 MB. yoloo | 05:57 |
| *** marios has joined #tripleo | 05:57 | |
| Tengu | hmmm, maybe I need some rebase of my code against master. | 05:58 |
| *** lifeless has quit IRC | 05:59 | |
| *** aditya_r has joined #tripleo | 06:00 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/puppet-tripleo master: Create dedicated "apache" base profile https://review.openstack.org/513274 | 06:00 |
| *** cshastri has quit IRC | 06:01 | |
| Tengu | couldn't do a proper rebase due to some conflict. might be the issue. | 06:01 |
| *** cshastri has joined #tripleo | 06:02 | |
| *** jprovazn has joined #tripleo | 06:07 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 06:10 |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 06:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 06:10 |
| *** aditya_r has quit IRC | 06:12 | |
| *** agopi_ has quit IRC | 06:13 | |
| *** dparkes has quit IRC | 06:13 | |
| *** yprokule has joined #tripleo | 06:14 | |
| *** brault has joined #tripleo | 06:20 | |
| *** jprovazn has quit IRC | 06:21 | |
| *** moshele has joined #tripleo | 06:21 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Add a tag to all the role specific parameters to identify and validate https://review.openstack.org/517231 | 06:23 |
| *** jprovazn has joined #tripleo | 06:28 | |
| *** brault has quit IRC | 06:35 | |
| *** moshele has quit IRC | 06:38 | |
| *** paramite has joined #tripleo | 06:40 | |
| *** aufi has joined #tripleo | 06:44 | |
| *** brault has joined #tripleo | 06:45 | |
| *** jaosorior has joined #tripleo | 06:46 | |
| *** stendulker has joined #tripleo | 06:48 | |
| *** jtomasek has joined #tripleo | 06:50 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Add capability to configure simple-crypto backend for barbican https://review.openstack.org/517312 | 06:51 |
| *** threestrands has quit IRC | 06:53 | |
| *** dparkes has joined #tripleo | 06:54 | |
| *** moshele has joined #tripleo | 06:56 | |
| jaosorior | good morning folks! | 06:59 |
| *** anshul has joined #tripleo | 06:59 | |
| jaosorior | Tengu: seems the unit tests are broken from mthe apache patch http://logs.openstack.org/74/513274/12/check/legacy-puppet-tripleo-puppet-unit-4.8-centos-7/90e331a/job-output.txt.gz#_2017-11-03_06_30_11_571773 | 07:00 |
| janki | Hi. Is there a way I can edit a file after containarised service is deployed? | 07:03 |
| jaosorior | janki: what file? | 07:04 |
| jaosorior | janki: you can :D | 07:04 |
| jaosorior | janki: the files that puppet generates are in /var/lib/config-data/puppet-generated/<service name>/<some path>/<the file you're looking for> | 07:05 |
| jaosorior | janki: after editing the file you can restart the container | 07:05 |
| janki | jaosorior, I have to edit .cfg file in ODL container. I did try ssh'ing into it and editing via vi. But not sure if it will take effect | 07:05 |
| *** spectr has joined #tripleo | 07:05 | |
| janki | jaosorior, docker restart? | 07:07 |
| jaosorior | yep | 07:07 |
| *** pdeore_ has joined #tripleo | 07:07 | |
| jaosorior | of course assuming that ODL was done in the way the other services was done and it uses the stuff from the puppet-generated dir | 07:07 |
| janki | ya. it uses stuff from puppet-generated. | 07:08 |
| jaosorior | ok | 07:08 |
| Tengu | jaosorior: o_O | 07:09 |
| janki | jaosorior, but the files I want to edit are not puppet-generated. | 07:09 |
| Tengu | jaosorior: they passed last time, all I did today was a kind of rebase… what.the.actual.fuck | 07:09 |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 07:10 |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 07:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 07:10 |
| *** pdeore has quit IRC | 07:10 | |
| jaosorior | Tengu: well, the errors I saw there seemed related :/ | 07:10 |
| janki | jaosorior, can I make changes inside the container and restart service? will that be good? | 07:10 |
| jaosorior | janki: mayhaps | 07:10 |
| jaosorior | janki: you could try | 07:10 |
| Tengu | jaosorior: hmm. weird. well, re-running the units on my computer in order to check the current state. | 07:11 |
| jaosorior | bandini: hey, could you check out this backport https://review.openstack.org/#/c/517324/ ? | 07:17 |
| jaosorior | marios: ^^ | 07:17 |
| jaosorior | bandini: nevermind, forgot you're on PTO | 07:17 |
| marios | jaosorior: ack | 07:19 |
| Tengu | jaosorior: ok, I think I know what happened X(. I commited from another computer, and didn't (re)fetch my branch… pfffffff. | 07:20 |
| jaosorior | aaah shit | 07:20 |
| jaosorior | right | 07:20 |
| jaosorior | marios: also, if you have a chance https://review.openstack.org/#/q/topic:bug/1729293+status:open | 07:21 |
| jaosorior | ratailor: are the EC2 patches ready? | 07:24 |
| *** pcaruana has joined #tripleo | 07:24 | |
| jaosorior | jbadiapa: is this backport fine https://review.openstack.org/#/c/517497/2 ? | 07:25 |
| *** jfrancoa has joined #tripleo | 07:28 | |
| Tengu | -.- and my gf just killed the home laptop. great. | 07:28 |
| ratailor | jaosorior, yes, I have removed the WIP flag for reviews. | 07:31 |
| jaosorior | ratailor: excellent; will check it out | 07:31 |
| jaosorior | Tengu: dropped coffee? | 07:31 |
| ratailor | jaosorior, sure, | 07:31 |
| ratailor | jaosorior, Do we need to check whether packets are encrypted or not ? | 07:32 |
| jbadiapa | jaosorior, let me check it | 07:32 |
| Tengu | jaosorior: nope, powerdown, meaning no reboot possible due to full disk encryption… I just asked for a display opening, as it was just asleep. anyway… re-doing the correction, running units on the work laptop, and re-pushing. | 07:32 |
| jaosorior | ratailor: definitely something to check :D Also, you're missing the containerized ec2. Which needs an extra container. But you can do that in a subsequent commit. | 07:33 |
| Tengu | should dedicate one of my NUC for that kind of work in fact, always on, and centralized dev through SSH :] | 07:34 |
| *** ccamacho has joined #tripleo | 07:34 | |
| ratailor | jaosorior, I think these patches just configure the service to run over TLS, and if haproxy is working properly, the packets will be encrypted. right ? | 07:35 |
| ratailor | jaosorior, yes, I have that container patch ready, but I was waiting for the on-going patches to merge. otherwise I have to make a long list of depends-on patches. | 07:36 |
| jaosorior | ratailor: well, the TLS proxy is serving TLS only, so if you try to do unencrypted traffic it will just plain fail the handshake. | 07:36 |
| *** adac has joined #tripleo | 07:38 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/puppet-tripleo master: Create dedicated "apache" base profile https://review.openstack.org/513274 | 07:39 |
| ccamacho | morning folks! | 07:39 |
| Tengu | hopefully the last time I need to do anything on that review… almost a months it's stuck in the CI X( | 07:40 |
| jaosorior | ccamacho: morning! | 07:40 |
| jaosorior | Tengu: I feel your pain https://review.openstack.org/#/q/owner:jaosorior%2540redhat.com+status:open | 07:40 |
| Tengu | :D | 07:41 |
| ratailor | jaosorior, cool. apart from checking haproxy configuration and running tempest successfully, Have you done anything else to check TLS working properly for nova-metadata ? | 07:42 |
| ratailor | jaosorior, after deploying successfully, I have also check the configuration for ec2api, ec2api-metadata service as well and ran tempest successfully. | 07:43 |
| *** moshele has quit IRC | 07:45 | |
| jaosorior | jbadiapa: thanks for checking it out. | 07:46 |
| jaosorior | ratailor: you can try tcpdump. What I was doing is openssl s_client for each endpoint that I enable. | 07:47 |
| ratailor | jaosorior, sure, Thanks! | 07:47 |
| jbadiapa | jaosorior, one question, The patch is ok for me, but wouldn't be better to use the fluentd container image? | 07:48 |
| openstackgerrit | yatin proposed openstack/tripleo-quickstart-extras master: Add reference of image for pike release in rdocloud env https://review.openstack.org/509090 | 07:49 |
| jaosorior | jbadiapa: it would. But I don't know the status of it. Is that working in pike? | 07:50 |
| *** etingof has quit IRC | 07:51 | |
| *** moshele has joined #tripleo | 07:54 | |
| marios | jaosorior: ack | 07:55 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Allow users to specify the Introspection concurrency https://review.openstack.org/451228 | 07:56 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Migrate to the new Mistral context class https://review.openstack.org/506186 | 07:57 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-heat-templates master: Make KeystoneURL versionless https://review.openstack.org/515351 | 07:57 |
| jaosorior | marios: thanks for the reviews! | 07:59 |
| *** dsariel__ has joined #tripleo | 07:59 | |
| openstackgerrit | Juan Badia Payno proposed openstack/tripleo-heat-templates master: Add fluentd on scenario001-multinode-containers https://review.openstack.org/517567 | 08:02 |
| jaosorior | Tengu: I'm fine with a POC for one service and we can see what needs to be done for every other service and how to enable them easier. | 08:03 |
| Tengu | jaosorior: great :). Just have to chose a victim | 08:03 |
| jaosorior | Tengu: perhaps doing keystone or glance would be the easiest thing. | 08:03 |
| jbadiapa | jaosorior, first we need to have fluentd enable on master for scenario001-multinode-containers | 08:04 |
| jaosorior | jbadiapa: is the image available already? mandre had put a comment there that it wasn't. | 08:04 |
| Tengu | jaosorior: hmm ok. shall we ask the ML about the victim of choice? | 08:05 |
| marios | np jaosorior | 08:05 |
| *** etingof has joined #tripleo | 08:05 | |
| *** ramishra has quit IRC | 08:05 | |
| jaosorior | Tengu: if you want. honestly I would just go for a low hanging fruit and take it from there. | 08:05 |
| *** yamahata has quit IRC | 08:05 | |
| Tengu | jaosorior: fine for me. I'll check when I have a spare hours for that. | 08:05 |
| jaosorior | Tengu: thanks for checking it out | 08:06 |
| Tengu | no problem | 08:06 |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 08:10 |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 08:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 08:10 |
| *** tesseract has joined #tripleo | 08:10 | |
| *** ramishra has joined #tripleo | 08:11 | |
| *** cylopez has joined #tripleo | 08:12 | |
| *** pdeore__ has joined #tripleo | 08:12 | |
| *** etingof has quit IRC | 08:12 | |
| *** ebarrera has joined #tripleo | 08:13 | |
| *** ratailor is now known as ratailor|Lunch | 08:13 | |
| *** pdeore_ has quit IRC | 08:15 | |
| *** ykarel is now known as ykarel|lunch | 08:15 | |
| *** gkadam has joined #tripleo | 08:17 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Ceilometer] https://review.openstack.org/507796 | 08:19 |
| jaosorior | jfrancoa: how's the validation tasks going? ready for reviews? | 08:20 |
| jfrancoa | jaosorior: I'm just going to do the last change (hopefuly :) ), adding the verification to stop the service only if it was enabled. Once I add it back for all services it should be the definitive version (unless other comments require more changes) | 08:23 |
| jaosorior | jfrancoa: let me know when you need reviews | 08:24 |
| jfrancoa | jaosorior: sure, thanks a lot | 08:24 |
| *** ffiore has joined #tripleo | 08:25 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Only mount Haproxy TLS if contents are set https://review.openstack.org/517173 | 08:25 |
| *** sri_ has quit IRC | 08:32 | |
| *** sri_ has joined #tripleo | 08:33 | |
| *** moshele has quit IRC | 08:34 | |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates stable/pike: Fix iptables rules override bug in clustercheck docker service https://review.openstack.org/517576 | 08:37 |
| *** amoralej|off is now known as amoralej | 08:38 | |
| sri_ | shardy: quick question, to use cinder-volume encryption I've enabled barbican is that enough or do I need pass any extra configuration? I've tried by jest enabling barbican volume encryption didn't work | 08:38 |
| *** ramishra has quit IRC | 08:39 | |
| *** ramishra has joined #tripleo | 08:41 | |
| dciabrin | mandre, hey how would I tell gate job for https://review.openstack.org/#/c/515326/1 has stalled? | 08:44 |
| lvdombrkr | hello folks... i have question about removing nodes from overcloud, if have 3 nodes controller HA setup, can i remove 2 controllers, and stay on 1? i mean without destroying existing overcloud ) | 08:45 |
| mandre | dciabrin: it's in the queue | 08:45 |
| mandre | dciabrin: you can see it at http://zuulv3.openstack.org/ | 08:46 |
| dciabrin | mandre, oh awesome, thx! | 08:46 |
| d0ugal | apetrich: FYI https://review.openstack.org/517577 | 08:47 |
| apetrich | d0ugal, oh nice | 08:48 |
| *** jpena|off is now known as jpena | 08:49 | |
| d0ugal | apetrich: almost nice, seems I got a syntax error >_< | 08:50 |
| d0ugal | actually, re-reading the error - it might be okay | 08:51 |
| d0ugal | ¯\_(ツ)_/¯ | 08:51 |
| *** hewbrocca_afk is now known as hewbrocca | 08:57 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Remove unused pre_network configuration https://review.openstack.org/517582 | 09:01 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Add a tag to all the role specific parameters https://review.openstack.org/517231 | 09:08 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 09:10 |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 09:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 09:10 |
| *** jpich has joined #tripleo | 09:12 | |
| *** ykarel|lunch is now known as ykarel | 09:12 | |
| *** pdeore__ is now known as pdeore | 09:13 | |
| *** ratailor|Lunch is now known as ratailor | 09:13 | |
| *** derekh has joined #tripleo | 09:14 | |
| *** jtomasek has quit IRC | 09:15 | |
| *** aditya_r has joined #tripleo | 09:20 | |
| *** lucas-afk is now known as lucasagomes | 09:21 | |
| *** jtomasek has joined #tripleo | 09:24 | |
| *** aditya_r has quit IRC | 09:24 | |
| *** aditya_r has joined #tripleo | 09:25 | |
| *** aditya_r has quit IRC | 09:27 | |
| *** aditya_r has joined #tripleo | 09:28 | |
| *** ratailor is now known as ratailor|session | 09:31 | |
| *** aditya_r has quit IRC | 09:32 | |
| *** ykarel is now known as ykarel|session | 09:32 | |
| *** pdeore_ has joined #tripleo | 09:36 | |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-common stable/pike: Fix typos in container_images_prepare failure task names https://review.openstack.org/517590 | 09:37 |
| *** pdeore has quit IRC | 09:39 | |
| *** mcornea has joined #tripleo | 09:39 | |
| *** udesale__ has joined #tripleo | 09:39 | |
| *** udesale has quit IRC | 09:41 | |
| *** jaganathan has quit IRC | 09:42 | |
| *** bogdando has joined #tripleo | 09:43 | |
| bogdando | o/ https://review.openstack.org/#/c/513669/ PTAL folks! | 09:52 |
| *** pradk has joined #tripleo | 09:57 | |
| *** links has quit IRC | 09:58 | |
| *** daidv has quit IRC | 10:10 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 10:10 |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 10:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 10:10 |
| Tengu | soooo… anyone for some +2 voting on that review? https://review.openstack.org/#/c/513274/ (once again, yes, sorry - it was a rebase with conflict) | 10:13 |
| *** udesale__ has quit IRC | 10:16 | |
| *** udesale has joined #tripleo | 10:16 | |
| openstackgerrit | Beth Elwell proposed openstack/tripleo-ui stable/pike: Exclude outdated validation results from list https://review.openstack.org/517597 | 10:17 |
| jaosorior | mandre: should I merge this https://review.openstack.org/#/c/502843/ ? | 10:19 |
| *** rcernin has joined #tripleo | 10:21 | |
| mandre | jaosorior: yeah, otherwise we'll lose the healthchecks when we get the containers promotion, maybe that's even the case as we speak | 10:21 |
| *** ckyriakidou has joined #tripleo | 10:23 | |
| *** ckyriakidou has quit IRC | 10:23 | |
| *** ckyriakidou has joined #tripleo | 10:23 | |
| jaosorior | mandre: ok, I'll merge it | 10:24 |
| *** pdeore__ has joined #tripleo | 10:25 | |
| *** ffiore has quit IRC | 10:26 | |
| *** pdeore_ has quit IRC | 10:28 | |
| *** pgadiya has joined #tripleo | 10:31 | |
| *** udesale__ has joined #tripleo | 10:32 | |
| *** ffiore has joined #tripleo | 10:33 | |
| *** udesale has quit IRC | 10:34 | |
| Tengu | … err 143… | 10:36 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Make kolla set permissions for haproxy public cert https://review.openstack.org/517599 | 10:39 |
| derekh | Where does this log file get collected? http://logs.openstack.org/68/517368/1/check-tripleo/legacy-tripleo-ci-centos-7-ovb-containers-oooq/e4eb712/logs/overcloud-controller-0/var/ | 10:44 |
| derekh | I'm looking to add /var/lib/config-data/puppet-generated/ | 10:44 |
| jaosorior | jistr: are you acquainted with the networking environments? we need to add an extra IP address for the containerized undercloud | 10:46 |
| jaosorior | pradk: ^^ | 10:46 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Neutron] https://review.openstack.org/489957 | 10:46 |
| jistr | jaosorior: not too much... what's the issue? | 10:47 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Heat] https://review.openstack.org/491399 | 10:48 |
| jaosorior | jistr: we're trying to do TLS for the containerized undercloud. And need another IP address from HAProxy to listen on | 10:48 |
| jaosorior | jistr: else we get port conflicts. | 10:48 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Aodh] https://review.openstack.org/491411 | 10:48 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Gnocchi] https://review.openstack.org/491433 | 10:48 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Mistral] https://review.openstack.org/493471 | 10:48 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Mixed-1] https://review.openstack.org/495195 | 10:49 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Nova] https://review.openstack.org/499108 | 10:49 |
| *** rfolco has joined #tripleo | 10:51 | |
| jistr | jaosorior: right i see. I don't have an overcloud right now but i think we solve that in overcloud by HAProxy listening on VIP, don't we? | 10:51 |
| *** achadha has joined #tripleo | 10:51 | |
| jaosorior | jistr: right; we just don't know how to do that via heat environments for the undercloud | 10:52 |
| jaosorior | jistr: in the classic undercloud we have two IP addresses that we assign to HAProxy. We just don't know how to do that via t-h-t. Like, assigning a pre-defined IP with the right networking template. | 10:53 |
| *** udesale__ has quit IRC | 10:54 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Ironic] https://review.openstack.org/502037 | 10:54 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Octavia] https://review.openstack.org/502038 | 10:55 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Sahara] https://review.openstack.org/502039 | 10:55 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Manila] https://review.openstack.org/502040 | 10:55 |
| *** achadha has quit IRC | 10:56 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Ovn] https://review.openstack.org/502041 | 10:56 |
| *** jprovazn has quit IRC | 10:59 | |
| jistr | jaosorior: i'm not familiar with that either but trying to trace the code paths | 11:02 |
| jistr | https://github.com/openstack/tripleo-heat-templates/blob/3b17b3fccdb4d8e0064ceec0cab2857d0c3cc7f0/puppet/all-nodes-config.j2.yaml#L178-L179 | 11:02 |
| jistr | https://github.com/openstack/tripleo-heat-templates/blob/3b17b3fccdb4d8e0064ceec0cab2857d0c3cc7f0/overcloud.j2.yaml#L766 | 11:02 |
| jistr | https://github.com/openstack/tripleo-heat-templates/blob/3b17b3fccdb4d8e0064ceec0cab2857d0c3cc7f0/overcloud.j2.yaml#L713-L743 | 11:02 |
| jistr | https://github.com/openstack/tripleo-heat-templates/blob/3b17b3fccdb4d8e0064ceec0cab2857d0c3cc7f0/overcloud.j2.yaml#L89-L103 | 11:03 |
| jistr | jaosorior: so setting ControlFixedIPs could be the trick. I *think* PublicVirtualIP would fall back to ControlVirtualIP if not set. | 11:04 |
| jistr | jaosorior: autoselection via neutron is not an option since we don't have any neutron at that point https://github.com/openstack/tripleo-heat-templates/blob/3b17b3fccdb4d8e0064ceec0cab2857d0c3cc7f0/overcloud.j2.yaml#L718 | 11:06 |
| *** stendulker_ has joined #tripleo | 11:06 | |
| * jistr grabs lunch with folks, biab | 11:06 | |
| jaosorior | jistr: right :/ we don't have neutron | 11:07 |
| jistr | so this one https://github.com/openstack/tripleo-heat-templates/blob/3b17b3fccdb4d8e0064ceec0cab2857d0c3cc7f0/overcloud.j2.yaml#L719 | 11:07 |
| jistr | it takes priority over neutron | 11:07 |
| *** psachin has quit IRC | 11:07 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Make kolla set permissions for haproxy public cert https://review.openstack.org/517599 | 11:09 |
| *** stendulker has quit IRC | 11:09 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 11:10 |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 11:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 11:10 |
| openstackgerrit | Pradeep Kilambi proposed openstack/python-tripleoclient master: Support to drive undercloud deploy via undercloud.conf https://review.openstack.org/511350 | 11:10 |
| *** stendulker_ has quit IRC | 11:10 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Swift] https://review.openstack.org/502042 | 11:10 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Mixed-2] https://review.openstack.org/505586 | 11:11 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Mixed-3] https://review.openstack.org/505587 | 11:11 |
| jfrancoa | jaosorior: marios: ccamacho: chem: the validation tasks for upgrading are done waiting for last CI results, any comments/reviews are appreciated. thanks a lot! https://review.openstack.org/#/q/status:open+project:openstack/tripleo-heat-templates+branch:master+topic:bug/1704389 | 11:16 |
| ccamacho | jfrancoa ack | 11:16 |
| marios | jfrancoa: ack thanks | 11:17 |
| *** ansmith has quit IRC | 11:17 | |
| jaosorior | jfrancoa: these https://review.openstack.org/#/q/topic:bug/1704389+status:open , right? | 11:19 |
| openstackgerrit | Keith Schincke proposed openstack/puppet-tripleo master: Set ACLs on ceph client keyrings https://review.openstack.org/517347 | 11:19 |
| jfrancoa | jaosorior: yes, exactly those ones | 11:19 |
| jaosorior | cool cool | 11:20 |
| *** anilvenkata has quit IRC | 11:20 | |
| *** ffiore has quit IRC | 11:21 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Mistral] https://review.openstack.org/493471 | 11:23 |
| *** ffiore has joined #tripleo | 11:24 | |
| *** atoth has joined #tripleo | 11:27 | |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Re-enable containerized fluentd https://review.openstack.org/517605 | 11:29 |
| *** dsneddon has quit IRC | 11:32 | |
| jaosorior | mandre: seems jbadiapa was doing that already https://review.openstack.org/#/c/517567/ | 11:32 |
| mandre | jaosorior: bah... | 11:35 |
| jaosorior | mandre: we could merge whichever | 11:35 |
| jaosorior | I don't mind | 11:35 |
| mandre | jaosorior: seems like jbadiapa submitted his before we had promotion... I'll wait for the other one to finish running the jobs | 11:36 |
| mandre | jaosorior: according to https://hub.docker.com/r/tripleomaster/centos-binary-fluentd/tags/ the promotion happened two hours ago | 11:37 |
| jaosorior | mandre: ack | 11:37 |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient master: Add --config-download https://review.openstack.org/514701 | 11:39 |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient master: Add --output-dir https://review.openstack.org/514789 | 11:39 |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart master: fs010: use --config-download from tripleoclient https://review.openstack.org/514736 | 11:40 |
| *** janki has quit IRC | 11:45 | |
| ccamacho | hey d0ugal thrash around ?? quick/basic question, should 'sudo mistral-db-manage populate' read the tripleo-common->setup.cfg file to load my new actions? | 11:47 |
| ccamacho | Im moving some UC backup actions to mistral but Im getting http://paste.openstack.org/show/625419/ any idea? | 11:47 |
| d0ugal | ccamacho: sort-of, it doesn't read the file directly. You need to install pip tripleo-common to have the entrypoints updated | 11:49 |
| d0ugal | ccamacho: do you have a patch up I can quickly check over? | 11:49 |
| *** shreshtha has quit IRC | 11:49 | |
| d0ugal | ccamacho: I don't follow these steps, but they should do everything you need: https://github.com/openstack/tripleo-common#action-development | 11:50 |
| d0ugal | ccamacho: the Python install, mistral restart and db populate are the key steps. It is a bit of a pain. | 11:50 |
| d0ugal | ccamacho: so you may just need to restart mistral if you have done the other bits. | 11:51 |
| * d0ugal really wants to improve this process | 11:51 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-common master: WIP add UC backup actions https://review.openstack.org/517610 | 11:53 |
| ccamacho | d0ugal thanks! I was kinda applying the changes ^ | 11:54 |
| *** abregman|afk is now known as abregman | 11:55 | |
| d0ugal | ccamacho: Nothing obvious jumps out at me. so if the above doesn't help we might need to dig deeper. | 11:56 |
| ccamacho | g0ugla np man Ill dig it thanks for the docs hint if seems OK the code its something in my env thanks! | 11:56 |
| * d0ugal assumes that was for him | 11:57 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/python-tripleoclient stable/pike: Add missing puppet configuration to undercloud deploy https://review.openstack.org/517613 | 11:57 |
| d0ugal | ccamacho: if you don't have any luck, and it is an env I can access, I'd be happy to take a look. | 11:57 |
| *** eck` is now known as eck`gone | 12:00 | |
| *** abishop has joined #tripleo | 12:04 | |
| ccamacho | d0ugal++ :) https://paste.fedoraproject.org/paste/JpK1sWULOPoHT6rRR2KwIQ I was missing the endpoints update https://github.com/openstack/tripleo-common#action-development was really useful Im not installing new packages because im hacking the config to point to my upstream review thanks! | 12:07 |
| *** rhallisey has joined #tripleo | 12:08 | |
| d0ugal | ccamacho: great. Hat tip to rbrady for adding that to the README originally (iirc) | 12:08 |
| *** raildo has joined #tripleo | 12:08 | |
| *** jlabarre has quit IRC | 12:08 | |
| *** ykarel|session is now known as ykarel | 12:09 | |
| *** ratailor|session has quit IRC | 12:09 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 12:10 |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 12:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 12:10 |
| *** jcoufal has joined #tripleo | 12:11 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Tell HAProxy container template that the public cert was autogenerated https://review.openstack.org/517615 | 12:13 |
| d0ugal | jpich: oh man, 1729835 is good | 12:14 |
| d0ugal | or terrible | 12:14 |
| d0ugal | maybe both | 12:14 |
| *** raildo has quit IRC | 12:14 | |
| *** raildo has joined #tripleo | 12:15 | |
| *** jlabarre has joined #tripleo | 12:15 | |
| jpich | d0ugal: I vote for both! I'm sure it's something silly and probably easy to fix, just don't have time to look into it straightaway so I opened the bug for now | 12:16 |
| d0ugal | jpich: good plan, thanks for reporting it | 12:16 |
| jpich | I just can't figure out why introspection is failing, can't find error messages anywhere | 12:16 |
| d0ugal | I rarely see the introspection output now, I should run it manually more often. | 12:17 |
| jpich | d0ugal: Np! | 12:17 |
| *** anshul has quit IRC | 12:17 | |
| *** janki has joined #tripleo | 12:18 | |
| *** leitan has joined #tripleo | 12:19 | |
| *** leitan_ has joined #tripleo | 12:20 | |
| *** leitan has quit IRC | 12:20 | |
| mandre | it seems the new containers are all failing tempest with a keystone v2 error :-/ | 12:20 |
| mandre | http://logs.openstack.org/09/514809/2/gate/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-container/ccf39c8/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2017-11-03_11_30_51 | 12:20 |
| *** lucasagomes is now known as lucas-hungry | 12:21 | |
| *** ebarrera has quit IRC | 12:21 | |
| *** jprovazn has joined #tripleo | 12:21 | |
| * dtantsur hears introspection and runs away | 12:23 | |
| *** ebarrera has joined #tripleo | 12:23 | |
| jaosorior | mandre: that should have been fixed already by a newer tempest config | 12:23 |
| jaosorior | seems that it's using an older package | 12:23 |
| jaosorior | chandankumar: can you check this out? http://logs.openstack.org/09/514809/2/gate/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-container/ccf39c8/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2017-11-03_11_30_51 | 12:24 |
| jistr | jaosorior: looked further and i see that undercloud.yaml overwrites the port resource (i was mistaken that the *FixedIPs doesn't use Neutron, it does use Neutron to reserve the IP from further allocation, even though Neutron doesn't make the decision what the IP address is) | 12:24 |
| jistr | jaosorior: https://github.com/openstack/tripleo-heat-templates/blob/master/deployed-server/deployed-neutron-port.yaml | 12:24 |
| mandre | jaosorior, chandankumar: is it something that can be fixed without rebuilding the containers? | 12:24 |
| jaosorior | no idea | 12:25 |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: WIP: Upgrade UC and OC using tripleo-upgrade role https://review.openstack.org/515643 | 12:25 |
| jaosorior | mandre: it should be available in a newer package | 12:25 |
| jaosorior | if the image contains too old of a package it has to be rebuilt | 12:26 |
| mandre | crap | 12:26 |
| jaosorior | lets see what chandankumar says | 12:26 |
| mandre | panda|ruck: you around? | 12:26 |
| panda|ruck | mandre: yes | 12:27 |
| panda|ruck | mandre: unless you have bad news | 12:27 |
| panda|ruck | mandre: then I'm not | 12:27 |
| mandre | panda|ruck: soooo | 12:27 |
| jistr | jaosorior: so i think we need DeployedServerPortMap parameter to include control_virtual_ip entry | 12:27 |
| *** ffiore has quit IRC | 12:27 | |
| mandre | it looks like tempest fails with the new containers | 12:27 |
| mandre | panda|ruck: https://bugs.launchpad.net/tripleo/+bug/1729848 | 12:27 |
| openstack | Launchpad bug 1729848 in tripleo "Tempest fails in container jobs trying to reach a keystone v2 URI" [Critical,Confirmed] | 12:27 |
| jaosorior | jistr: ok. Any hints on how to use that? | 12:28 |
| therve | mandre, Yeah just saw that one :/ | 12:29 |
| jistr | jaosorior: here's the param format but i don't know how parameter_defaults are best passed to undercloud deploy https://github.com/openstack/tripleo-heat-templates/blob/3b17b3fccdb4d8e0064ceec0cab2857d0c3cc7f0/deployed-server/deployed-neutron-port.yaml#L10-L15 | 12:29 |
| jistr | i don't deploy containerized UC nowadays | 12:30 |
| jistr | jaosorior: do you use the undercloud deploy command? it looks like it should accept -e same way as overcloud deploy: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/undercloud_deploy.py#L442 | 12:32 |
| *** lblanchard has joined #tripleo | 12:32 | |
| jistr | jaosorior: oh i see the problem | 12:34 |
| jistr | https://github.com/openstack/python-tripleoclient/blob/79cbcddd7e16d29cb284617fc039cf50ad2b07ed/tripleoclient/v1/undercloud_deploy.py#L341-L350 | 12:34 |
| *** ecerquei has joined #tripleo | 12:34 | |
| jaosorior | pradk: ^^ | 12:35 |
| jistr | we can't feed the same address for both IP and VIP | 12:36 |
| jaosorior | right, I guess we need to change that | 12:36 |
| jaosorior | and also set the public_virtual_ip there. | 12:37 |
| *** leitan_ has quit IRC | 12:37 | |
| *** catintheroof has joined #tripleo | 12:37 | |
| *** catintheroof has quit IRC | 12:37 | |
| *** catintheroof has joined #tripleo | 12:37 | |
| panda|ruck | mandre: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/e46795b/undercloud/home/jenkins/tempest_output.log.txt.gz | 12:38 |
| panda|ruck | mandre: we tested featuresets16-19 with tempest, above are the results | 12:38 |
| jprovazn | jaosorior: hi | 12:39 |
| mandre | panda|ruck: ack, i'll keep looking | 12:39 |
| *** jcoufal_ has joined #tripleo | 12:39 | |
| ykarel | mandre, tempest runs in container? | 12:39 |
| weshay | mandre, that was validated w/ https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/e82a337/ | 12:39 |
| mandre | ykarel: no! it just occurred to me as well | 12:39 |
| *** pdeore__ has quit IRC | 12:40 | |
| ykarel | mandre, also chandankumar is on pycon summit so might not be available sooner | 12:40 |
| weshay | mandre, how consistently are you seeing the issue? | 12:40 |
| mandre | ykarel: ok good to know, will you be able to help on tempest config if needed? | 12:41 |
| jaosorior | jprovazn: hello | 12:41 |
| *** jcoufal has quit IRC | 12:41 | |
| ykarel | mandre, i don't have much idea, but i can check if i can find something | 12:41 |
| mandre | weshay: idk, the first jobs using the container just came back | 12:41 |
| mandre | weshay: could it be a wrong config in the scenario files? | 12:42 |
| weshay | mandre, k which review? | 12:42 |
| *** pgadiya has quit IRC | 12:42 | |
| weshay | mandre, no.. the same job passed w/ the same config, same containers | 12:42 |
| jprovazn | jaosorior: I have a question to templates yam validations - specifically to https://github.com/openstack/tripleo-heat-templates/blob/master/tools/yaml-validate.py#L44 | 12:42 |
| weshay | that's how it was promoted :) | 12:42 |
| mandre | weshay: https://review.openstack.org/#/c/514809/ | 12:42 |
| weshay | but we need to look at what's going | 12:42 |
| weshay | on | 12:42 |
| jaosorior | jprovazn: what's up? | 12:42 |
| mandre | weshay: the result don't yet show up in gerrit, but you'll see them in zuul | 12:43 |
| jprovazn | jaosorior: I have a use case where I use step_config in containers - https://review.openstack.org/#/c/471245/6/docker/services/ceph-ansible/ceph-nfs.yaml | 12:43 |
| jprovazn | jaosorior: this one is failing now because docker_config is missing in the template, but if I add it, then step_config is ignored because: | 12:43 |
| jprovazn | https://github.com/openstack/tripleo-heat-templates/blob/master/common/services.yaml#L72 | 12:43 |
| mandre | weshay: interesting, scenario003-container is green | 12:44 |
| *** ffiore has joined #tripleo | 12:44 | |
| jprovazn | jaosorior: so it seems to me (only based on brief check) that I should either do something differently in the template or we should allow missing docker_config under some cicumstances in validation | 12:45 |
| dciabrin | mandre, if you have a minute, https://review.openstack.org/#/c/517576 is lacking one +2/+A | 12:45 |
| dciabrin | mandre, awesome thx! | 12:46 |
| jaosorior | jprovazn why not separate the template? make puppet/services/ceph-nfs.yaml, and then do the docker/services/ file like you would usually od | 12:46 |
| jaosorior | *do | 12:46 |
| *** gkadam has quit IRC | 12:47 | |
| *** jcoufal_ has quit IRC | 12:47 | |
| jprovazn | jaosorior: you mean splitting configuration of single service into both parts? | 12:47 |
| weshay | mandre, ya.. the upstream job has an old version of tempest | 12:48 |
| weshay | @delorean | 12:48 |
| weshay | openstack-tempest.noarch 1:17.0.1-0.20170908221615.fc28933.el7.centos | 12:48 |
| weshay | vs.. what was promoted | 12:48 |
| *** jcoufal has joined #tripleo | 12:48 | |
| weshay | openstack-tempest.noarch 1:17.1.1-0.20171031061321.8f7915c.el7.centos | 12:48 |
| *** ansmith has joined #tripleo | 12:48 | |
| weshay | mandre, that was not a promoted build afaict | 12:48 |
| weshay | panda|ruck, ^ | 12:49 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart-extras master: Switch to RDO Queens baseurl in overcloud upgrade https://review.openstack.org/517280 | 12:49 |
| weshay | mandre, see https://trunk.rdoproject.org/centos7-master/current-tripleo/ | 12:49 |
| *** aputtur_ has joined #tripleo | 12:49 | |
| jaosorior | jprovazn: yep | 12:50 |
| mandre | weshay: so... could be that this job installed the undercloud prior to promotion and downloaded the containers just after the promotion? | 12:50 |
| weshay | mandre, ya.. who knows.. updating repos is not coordinated w/ jobs :) | 12:50 |
| weshay | rasca, merged :) | 12:51 |
| *** toure_biab is now known as toure | 12:51 | |
| rasca | weshay, thanks weshay! | 12:52 |
| jprovazn | jaosorior: to clarify, this ceph-nfs service is specific - it's configured by ceph-ansible, but then puppet takes care of creating pacemaker resource for it in later phase, splitting configuration into 2 templates (when I need both at the same time seems to me as a workaround), wouldn't it make more sense to keep the logic in the same place? | 12:52 |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart-extras master: config-download: use workflow supported in Queens https://review.openstack.org/516841 | 12:52 |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart master: fs010: use --config-download from tripleoclient https://review.openstack.org/514736 | 12:52 |
| mandre | weshay: bingo, the successful scenario003-containers job was using openstack-tempest-17.1.1-0.20171031061321.8f7915c.el7.centos.noarch | 12:53 |
| *** lucas-hungry is now known as lucasagomes | 12:54 | |
| *** rcernin has quit IRC | 12:54 | |
| mandre | weshay: that confirms the theory that scenario 1, 2, and 4 had an undercloud installed just before the promotion | 12:54 |
| *** abregman is now known as abregman|afk | 12:54 | |
| weshay | rasca are you testing rdo on centos or rdo on rhel | 12:55 |
| jaosorior | jprovazn: well, I guess then the easiest would be to add an exception in the validation | 12:55 |
| weshay | see matt's email | 12:55 |
| jaosorior | jprovazn: no biggie | 12:55 |
| rasca | weshay, rdo on centos | 12:55 |
| rasca | weshay, osp on rhel | 12:55 |
| rasca | weshay, saw the email and got confused. I was using the [2] address that myoung|bbl mentioned, but then everything broke | 12:56 |
| *** jpena is now known as jpena|lunch | 12:57 | |
| rasca | weshay, then I read your answer and corrected my jobs | 12:57 |
| weshay | rasca, wait.. confirm the url.. /me responding | 12:57 |
| openstackgerrit | Andy Smith proposed openstack/tripleo-heat-templates master: WIP Separate rpc and notify messaging backends https://review.openstack.org/507963 | 12:59 |
| jprovazn | jaosorior: ok, cool, I will include it | 13:00 |
| *** myoung|bbl is now known as myoung | 13:00 | |
| *** dougbtv_ has joined #tripleo | 13:00 | |
| *** yamahata has joined #tripleo | 13:00 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Add critical section for zypper https://review.openstack.org/517620 | 13:02 |
| janki | jaosorior, hey | 13:03 |
| *** jcoufal has quit IRC | 13:05 | |
| *** jmelvin has joined #tripleo | 13:06 | |
| *** Dinesh_Bhor has quit IRC | 13:08 | |
| *** jcoufal has joined #tripleo | 13:08 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 13:10 |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 13:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 13:10 |
| sri_ | hello folks: is cinder volume encryption supports in tripleo ? I've created overcloud by enabling barbican -e environments/services/barbican.yaml, but when I am triing to create encrypted volume I getting this error | 13:11 |
| sri_ | openstack volume create --size 1 --type LUKS 'encrypted volume' | 13:11 |
| sri_ | The server has either erred or is incapable of performing the requested operation. (HTTP 500) | 13:11 |
| *** udesale has joined #tripleo | 13:11 | |
| mandre | panda|ruck, jaosorior, therve: we figured what happened with the tempest failure, the undercloud was using packages from prior the promotion | 13:12 |
| jaosorior | ah | 13:12 |
| jaosorior | mandre: ok ok | 13:13 |
| jaosorior | janki: sup? | 13:13 |
| *** jmelvin_ has joined #tripleo | 13:13 | |
| jaosorior | mandre: well, then, we're all good then, right? :D | 13:13 |
| jaosorior | with the promotion | 13:13 |
| janki | jaosorior, I have few questions on overcloud | 13:13 |
| janki | jaosorior, good time? | 13:13 |
| jaosorior | sure | 13:13 |
| *** ffiore has quit IRC | 13:13 | |
| lvdombrkr | hello folks... i have question about removing nodes from overcloud, if have 3 nodes controller HA setup, can i remove 2 controllers, and stay on 1? i mean without destroying existing overcloud ) | 13:13 |
| mandre | jaosorior: I think so yeah | 13:14 |
| *** cdearborn has joined #tripleo | 13:14 | |
| janki | jaosorior, so 1 - I want to do sudo operations inside the container. where can I find the password? | 13:14 |
| jaosorior | no idea | 13:15 |
| *** jmelvin has quit IRC | 13:15 | |
| *** moshele has joined #tripleo | 13:18 | |
| jaosorior | janki: not even sure if there is sudo in the containers (there shouldn't be) | 13:19 |
| *** moshele has quit IRC | 13:21 | |
| *** jcoufal has quit IRC | 13:22 | |
| *** jcoufal has joined #tripleo | 13:22 | |
| sri_ | lvdombrkr: I don't think that's possible to do that , at least there is no easy way to do it . https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/9/html/director_installation_and_usage/sect-scaling_the_overcloud | 13:22 |
| *** lucasagomes is now known as lucas-rellocate | 13:23 | |
| sai | jaosorior: janki i just did docker exec -it keystone /bin/bash and i got in as root :/ | 13:26 |
| ccamacho | d0ugal :P sorry man for bugging you, now after executing: | 13:27 |
| ccamacho | (undercloud) [stack@undercloud ~]$ mistral run-action tripleo.undercloud.get_free_space | 13:27 |
| ccamacho | Im getting "ImportError: No module named tripleo_common.actions.undercloud" I assume this is because Mistral is not able to get "tripleo-common/tripleo_common/actions/undercloud.py" | 13:27 |
| ccamacho | But the entry_points are OK I have the action loaded.. so any idea about what to check? | 13:27 |
| ykarel | sai, because keystone container is running as root: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/keystone.yaml#L132 | 13:28 |
| *** skramaja has quit IRC | 13:28 | |
| panda|ruck | mandre: great | 13:28 |
| *** trown|outtypewww is now known as trown | 13:28 | |
| ykarel | hmm wrong reference, fetching correct one | 13:29 |
| *** udesale has quit IRC | 13:30 | |
| janki | sai, maybe because keystone needs to be run as root | 13:30 |
| fultonj | mandre: would you mind having a look at https://review.openstack.org/#/c/508975 ? | 13:30 |
| *** ffiore has joined #tripleo | 13:31 | |
| d0ugal | ccamacho: that is odd. Did you restart Mistral? | 13:31 |
| fultonj | mandre: in my Oct 26 11:03 comment i point out that nova-compute didn't get the expected ownership from the kola directives (but glance did) do you see anyhting in my kola directives? | 13:31 |
| d0ugal | ccamacho: Can you import it in a Python REPL? Maybe there is an error. | 13:31 |
| janki | sai, there are some services that run some commands as root and change to <service> user back | 13:32 |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud stable/newton: undercloud/upgrade: change ownership for ironic-dbsync.log https://review.openstack.org/517356 | 13:32 |
| ccamacho | hey d0ugal I never did REPL with Mistral actions, that's new kung-fu for me, do you have any docs to point and Ill follow them | 13:34 |
| openstackgerrit | Jan Provaznik proposed openstack/tripleo-heat-templates master: Add support for ceph-nfs manila backend https://review.openstack.org/471245 | 13:34 |
| *** pkovar has joined #tripleo | 13:35 | |
| d0ugal | ccamacho: I'm not sure you can do everything, but it would at least verify that the importing works. i.e. just: "$ python" and then >>> from tripleo_common.actions import undercloud | 13:36 |
| chem | ccamacho: marios we should merge https://review.openstack.org/#/c/509718/ and https://review.openstack.org/#/c/512397/ when you have time (miss only W+1) | 13:37 |
| jaosorior | Hey folks! could I get reviews for this https://review.openstack.org/#/c/517349/2 ? | 13:38 |
| dtantsur | hi folks! a small and lonely ocata backport is eager for your attention :) https://review.openstack.org/#/c/510817/ | 13:39 |
| *** eck`gone is now known as eck` | 13:40 | |
| mandre | fultonj: let me take a look | 13:40 |
| fultonj | thanks | 13:40 |
| lvdombrkr | sri_: yes i read this documentation, discovering maybe some folks tried to do that ) | 13:41 |
| *** jprovazn has quit IRC | 13:41 | |
| d0ugal | jaosorior: you are a faster clicker than I :) | 13:41 |
| jaosorior | it's the coffee | 13:41 |
| d0ugal | dammit, I knew tea was a mistake | 13:41 |
| fultonj | :) | 13:42 |
| dtantsur | also some long reads related to ironic, needed to make people do right things for pike and ocata: https://review.openstack.org/#/q/status:open+project:openstack/tripleo-docs+branch:master+topic:ironic | 13:42 |
| janki | jaosorior, whats the work around to run sudo command then? | 13:43 |
| jaosorior | janki: what do you want to do? | 13:44 |
| janki | jaosorior, sudo perf record -F 99 -a -g -- sleep 30 | 13:44 |
| *** jcoufal_ has joined #tripleo | 13:45 | |
| *** jcoufal has quit IRC | 13:45 | |
| marios | chem: ack, for the internal api one i think it is good to go, the other one could maybe do with more recent ci run? | 13:45 |
| jaosorior | janki: the workaround is to do a docker exec over that container, and specify the user to be root | 13:45 |
| chem | marios recheck or rebase | 13:46 |
| janki | jaosorior, thought the same. trying it | 13:46 |
| chem | marios rebasing and let's see how it goes | 13:46 |
| chem | marios thanks (I rebased the first one) | 13:47 |
| janki | sai do sudo docker exec --user root -it opendaylight_api /bin/bash | 13:47 |
| mandre | fultonj: it's weird, do you have a live environment where I can check what's going on? | 13:48 |
| sai | janki: nice, thanks! | 13:48 |
| sai | janki: if ineed some linux tools like perf, is it based to make the image with it, or install it post deployment? | 13:49 |
| fultonj | mandre: yes i do | 13:49 |
| *** amoralej is now known as amoralej|lunch | 13:49 | |
| *** bobh has joined #tripleo | 13:50 | |
| janki | sai, install post deployment. if you do it dockerfile, you will have to rebuild the image and start the deployment all over again. also you cant custom build image downstream afaik | 13:51 |
| *** jcoufal_ has quit IRC | 13:51 | |
| *** jcoufal has joined #tripleo | 13:51 | |
| sai | janki: ack, thanks | 13:51 |
| janki | jaosorior, is there is an example for using overcloud container image build? I cant seem to figure out config-file content | 13:51 |
| jaosorior | mandre: ^^ | 13:52 |
| *** jkilpatr has quit IRC | 13:52 | |
| *** chlong has joined #tripleo | 13:55 | |
| *** pkovar has quit IRC | 13:56 | |
| *** ykarel is now known as ykarel|away | 13:58 | |
| *** jpena|lunch is now known as jpena | 14:01 | |
| mwhahaha | happy friday | 14:02 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common master: Updated from global requirements https://review.openstack.org/511633 | 14:02 |
| *** dsariel__ has quit IRC | 14:03 | |
| *** jcoufal_ has joined #tripleo | 14:04 | |
| *** ansmith has quit IRC | 14:04 | |
| EmilienM | hi | 14:05 |
| EmilienM | happy... friday :P | 14:05 |
| trown | happy birthday EmilienM | 14:05 |
| *** ykarel|away has quit IRC | 14:05 | |
| *** jcoufal has quit IRC | 14:06 | |
| marios | all the best happy birthday EmilienM (assuming trown isnt trolling) | 14:06 |
| EmilienM | how do you guys know | 14:06 |
| *** ansmith has joined #tripleo | 14:06 | |
| EmilienM | but thanks :-) I promess to be nice now I'm old | 14:06 |
| mwhahaha | pffft | 14:06 |
| EmilienM | (30 is old right?) | 14:06 |
| mwhahaha | i'm sure you'll be the epitome of the grump old man | 14:07 |
| trown | hahaha | 14:08 |
| EmilienM | it's probably in my dna | 14:08 |
| trown | "back in my day we had to fly our own planes... none of this fully autonomous hyperloop crap" | 14:08 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 14:10 |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 14:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 14:10 |
| *** jcoufal_ has quit IRC | 14:10 | |
| trown | 30 is a legit birthday though... you only get to celebrate every 10 years now | 14:11 |
| jpich | EmilienM, so old! Happy birthday :) | 14:11 |
| beagles | EmilienM, Happy Birthday!!! | 14:12 |
| *** jcoufal_ has joined #tripleo | 14:12 | |
| * beagles declares that 30 is still pretty young | 14:12 | |
| *** jkilpatr has joined #tripleo | 14:12 | |
| * beagles concedes from that is from his vantage point anyways | 14:12 | |
| trown | beagles: I am with you on that... oh to be 30 again :P | 14:13 |
| * EmilienM listen old people :P | 14:13 | |
| beagles | lol | 14:13 |
| *** bnemec is now known as beekneemech | 14:14 | |
| myoung | town: ping | 14:15 |
| *** zshi has quit IRC | 14:15 | |
| trown | myoung: that didnt work :P | 14:15 |
| myoung | trown: ya wrong channel lol | 14:15 |
| trown | myoung: also it is trown not town :P | 14:15 |
| myoung | s/town/trown: indeed | 14:16 |
| jaosorior | EmilienM: happy birthday dude :D | 14:17 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: Set file mode permission for Ceph keyrings in containers https://review.openstack.org/508975 | 14:21 |
| EmilienM | jaosorior: thanks :-) | 14:21 |
| mandre | EmilienM: can you have a look at http://paste.openstack.org/show/625434/? | 14:21 |
| EmilienM | does it mean you can +2 all my patches today? :P | 14:21 |
| EmilienM | mandre: lol | 14:22 |
| EmilienM | mandre: nice one :P | 14:22 |
| * mwhahaha gives EmilienM a happy birthday -2 | 14:22 | |
| EmilienM | I didn't expect more from you ahah :P | 14:22 |
| mandre | haha | 14:22 |
| *** lucas-rellocate is now known as lucasagomes | 14:25 | |
| *** jprovazn has joined #tripleo | 14:26 | |
| *** vipul_ has joined #tripleo | 14:27 | |
| *** vipul_ has quit IRC | 14:27 | |
| mwhahaha | weshay, adarazs|rover, panda|ruck: where are we at with promotions? | 14:28 |
| weshay | mwhahaha, panda|ruck sent an email to openstack-dev, I'm following up | 14:29 |
| weshay | master promoted | 14:29 |
| mwhahaha | k i hadn't gotten through my email yet, i shall read | 14:29 |
| weshay | mwhahaha, EmilienM do you guys have a link to etherpad for master issues by chance? | 14:29 |
| weshay | mwhahaha, some jobs are picking up old repos still and we're seeing some failures | 14:29 |
| mwhahaha | https://etherpad.openstack.org/p/tripleo-promotion-blockers-october-2017 | 14:30 |
| weshay | going to update the email w/ that info | 14:30 |
| mwhahaha | that one? | 14:30 |
| weshay | thanks | 14:30 |
| weshay | ya | 14:30 |
| jpich | mandre: Oh man I thought the balloons were meant to be the eyes of some kind of demented owl at first | 14:32 |
| ccamacho | d0ugal mmm I even rebooted the UC :P and I can import the python class, at least is available (http://paste.openstack.org/show/625433/).. but.. still "ImportError: No module named tripleo_common.actions.undercloud" so should I import it somewhere inside mistral to be able to run the action like "mistral run-action tripleo.undercloud.get_free_space" | 14:33 |
| jaosorior | larsks: does the sample rsyslog.conf you gave to output to stdout need to be listening on that port? https://github.com/larsks/ooo-syslog-example/blob/master/rsyslog-stdout/rsyslog.conf would it work if we just use imuxsock and not imudp ? | 14:34 |
| d0ugal | ccamacho: no, that is strange. I would check the output from the populate command - there will be lots of errors but see if there is one related to your action. | 14:34 |
| *** dsariel__ has joined #tripleo | 14:35 | |
| larsks | jaosorior: Yes, that should work fine. Using udp avoids path issues in chrooted environments, but isn't strictly necessary. | 14:36 |
| jaosorior | larsks: I hadn't thought of that. Good call | 14:37 |
| *** cshastri has quit IRC | 14:43 | |
| jschlueter | ok testing on pike and while installing undercloud ... openstack undercloud install ==> I see DEPRECATION warnings showing up ... [WARNING] DEPRECATED: falling back to /var/run/os-collect-config/os_config_files.json | 14:44 |
| *** moshele has joined #tripleo | 14:45 | |
| jschlueter | is this fixed in master or is there an outstanding launch pad for it? | 14:46 |
| mwhahaha | jschlueter: no idea, would have to look into that further but i'm not aware of that | 14:47 |
| beekneemech | I think that's an ooooold deprecation warning. Kind of like EmilienM. ;-) | 14:47 |
| mandre | jistr, jaosorior: let's bring back containerized fluentd, https://review.openstack.org/#/c/517605/ | 14:48 |
| beekneemech | I think we tried to move away from /var/run way back when I first started on TripleO but never really followed through on it. | 14:48 |
| jschlueter | mwhahaha: I think it's https://pypi.python.org/pypi/os-apply-config related to the deprecation they are indicating in the usage section | 14:48 |
| jschlueter | mwhahaha, beekneemech: hmm we might want to make sure we do before we hit another keystone v2 incident ... where we have to scramble to fix things after a deprecated feature is pruned | 14:49 |
| jschlueter | beekneemech: /me nods | 14:49 |
| *** amoralej|lunch is now known as amoralej | 14:49 | |
| mwhahaha | weshay, adarazs|rover, panda|ruck, EmilienM: FYI I think when failed jobs in the gate reach the top of the queue the jobs below it are getting reset again. I've mentioned it in #openstack-infra but it would be a good idea to keep an eye on this and be able to provide additional data | 14:49 |
| beekneemech | jschlueter: We're not going to to remove that feature without first fixing the usage. It's our project. We literally couldn't merge such a patch. :-) | 14:50 |
| weshay | mwhahaha, so.. I've noticed a change there.. the jobs in progress are no longer restarted | 14:50 |
| mwhahaha | jschlueter: we're trying to get rid of instack-undercloud this cycle and we own os-apply-config | 14:50 |
| weshay | afaict | 14:50 |
| mwhahaha | weshay: the top of teh gate was restarted | 14:50 |
| jschlueter | mwhahaha: ack cool | 14:50 |
| mwhahaha | weshay: and i think it was when the failed job reached teh top | 14:50 |
| * jschlueter on warning/ deprecation reporting kick this morning | 14:50 | |
| beekneemech | And given how long ago we deprecated that, we should maybe just remove the message and stop pretending we're ever going to fix it. | 14:50 |
| mwhahaha | see 515326,1, it was just started a few bits ago | 14:50 |
| *** achadha has joined #tripleo | 14:51 | |
| *** itlinux has joined #tripleo | 14:52 | |
| weshay | hrm.. openstack health is not updated nearly enough | 14:54 |
| mwhahaha | weshay: yea, jeblair is looking into it | 14:55 |
| weshay | cool | 14:55 |
| mwhahaha | weshay: we should keep track of the failed jobs (patch numbers) and the jobs after it so we can see if it's resetting | 14:55 |
| mwhahaha | i just noticed it, so the next time something fails in gate i'm going to collect this info | 14:55 |
| *** achadha has quit IRC | 14:56 | |
| weshay | mwhahaha, ok.. let's do that in yet another etherpad | 14:56 |
| weshay | actually | 14:57 |
| mwhahaha | now we just have to wait for a job to fail :D | 14:57 |
| weshay | ha ha.. | 14:57 |
| weshay | won't be long | 14:57 |
| weshay | mwhahaha, https://trello.com/c/XfIlAb5q/311-master-jobs-failing-in-the-gate | 14:59 |
| weshay | panda|ruck, ^ | 14:59 |
| janki | Hey, can I recreate a container on overcloud? | 14:59 |
| mwhahaha | weshay: k | 14:59 |
| janki | with redeployong overcloud | 14:59 |
| janki | s/with/without | 14:59 |
| EmilienM | beekneemech: lol, nice joke :P | 15:00 |
| EmilienM | mwhahaha: ack | 15:01 |
| *** jkilpatr has quit IRC | 15:01 | |
| mwhahaha | weshay: sounds like the job before it failed out so might be nothing | 15:01 |
| weshay | mwhahaha, ya | 15:02 |
| *** moshele has quit IRC | 15:06 | |
| openstackgerrit | Keith Schincke proposed openstack/puppet-tripleo master: Set ACLs on ceph client keyrings https://review.openstack.org/517347 | 15:09 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 15:10 |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 15:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 15:10 |
| beekneemech | Yikes, ovb-containers job is looking pretty red. | 15:11 |
| beekneemech | 401 Unauthorized: This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required. (HTTP 401) | 15:12 |
| *** jkilpatr has joined #tripleo | 15:14 | |
| *** aditya_r has joined #tripleo | 15:16 | |
| *** itlinux has quit IRC | 15:20 | |
| *** janki has quit IRC | 15:20 | |
| *** janki has joined #tripleo | 15:21 | |
| *** janki has quit IRC | 15:22 | |
| *** karthiks has quit IRC | 15:22 | |
| *** jprovazn has quit IRC | 15:26 | |
| *** dparkes has quit IRC | 15:27 | |
| *** jaosorior has quit IRC | 15:27 | |
| *** itlinux has joined #tripleo | 15:29 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Validate roles data and network data https://review.openstack.org/508567 | 15:31 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add a Get Networks workflow https://review.openstack.org/509419 | 15:31 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add UpdateNetworks workflow https://review.openstack.org/513463 | 15:31 |
| thrash | rbrady: ^^^ | 15:31 |
| openstackgerrit | Thiago da Silva proposed openstack/tripleo-quickstart master: fix indentation on getting started page https://review.openstack.org/517642 | 15:32 |
| *** ffiore has quit IRC | 15:34 | |
| *** links has joined #tripleo | 15:35 | |
| *** nguyentrihai has joined #tripleo | 15:38 | |
| *** ykarel|away has joined #tripleo | 15:39 | |
| mwhahaha | weshay, mandre, EmilienM, adarazs|rover, panda|ruck: FYI reverting container-undercloud voting https://review.openstack.org/#/c/517643/ | 15:43 |
| EmilienM | +1 | 15:43 |
| mwhahaha | 514576,1 failed on it in the gate again | 15:43 |
| mwhahaha | I would love to see it back voting but we need to track it's stability better | 15:44 |
| *** ffiore has joined #tripleo | 15:46 | |
| EmilienM | ++ | 15:47 |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: WIP: Upgrade UC and OC using tripleo-upgrade role https://review.openstack.org/515643 | 15:48 |
| *** spectr has quit IRC | 15:49 | |
| *** jcoufal has joined #tripleo | 15:50 | |
| openstackgerrit | Merged openstack/tripleo-ui stable/pike: Exclude outdated validation results from list https://review.openstack.org/517597 | 15:52 |
| *** jcoufal_ has quit IRC | 15:53 | |
| ccamacho | d0ugal thanks man, | 15:53 |
| ccamacho | (undercloud) [stack@undercloud tripleo-common]$ mistral run-action tripleo.undercloud.get_free_space | 15:53 |
| ccamacho | {"result": 23844204} | 15:53 |
| ccamacho | Yeeeii! | 15:53 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Allow containerized undercloud deploy with SELinux https://review.openstack.org/513669 | 15:55 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Allow SELinux shared context for containers state https://review.openstack.org/517383 | 15:57 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Test containers with SELinux Enforcing https://review.openstack.org/517384 | 16:00 |
| *** yamahata has quit IRC | 16:03 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 16:10 |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729881 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 16:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 16:10 |
| openstack | Launchpad bug 1729881 in tripleo "401 unauthorized in overcloud-validate on ovb-containers job" [Critical,Triaged] | 16:10 |
| d0ugal | ccamacho: nice! What fixed it? | 16:10 |
| ccamacho | d0ugal I restarted the UC and ran | 16:12 |
| ccamacho | for workbook in workbooks/*.yaml; do mistral workbook-update $workbook; done | 16:12 |
| ccamacho | After that the python issue went away :) again thanks for the hints :) | 16:12 |
| d0ugal | ccamacho: np! glad it worked eventually. | 16:12 |
| *** ykarel|away has quit IRC | 16:16 | |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-common master: Validate parameters before updating https://review.openstack.org/511249 | 16:17 |
| ansiwen | any haproxy experts around? | 16:18 |
| ansiwen | ec2-api returns a 400 to GET /, that seems to let haproxy think, it's not available | 16:18 |
| ansiwen | can I tell haproxy, that 400 is ok? or is that a bad thing to do? | 16:18 |
| mwhahaha | ansiwen: yea so it want's a proper 200 request | 16:18 |
| ansiwen | but ec2-api replies with 400 without authorization | 16:19 |
| mwhahaha | ansiwen: ec2-api really should have a health check url | 16:19 |
| mwhahaha | ansiwen: you could switch it to tcpchk rather than httpchk | 16:20 |
| mwhahaha | and work on getting a proper health check url that returns 2xx or 3xx | 16:20 |
| mwhahaha | ansiwen: https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#4-option%20httpchk | 16:21 |
| *** jcoufal has quit IRC | 16:21 | |
| ansiwen | I'm just googling for an healthcheck url | 16:21 |
| *** achadha has joined #tripleo | 16:21 | |
| mwhahaha | ansiwen: the ec2api needs to have a request that can be made w/o auth that response with a 2xx or 3xx | 16:22 |
| mwhahaha | ie /healtcheck | 16:22 |
| *** aditya_ra has joined #tripleo | 16:22 | |
| mwhahaha | that we would configure haproxy to check | 16:22 |
| *** achadha_ has joined #tripleo | 16:22 | |
| mwhahaha | ansiwen: for example we check that /v3 is available for keystone https://github.com/openstack/puppet-tripleo/blob/master/manifests/haproxy.pp#L879 | 16:23 |
| ansiwen | mwhahaha: "needs" as in that is probably there, I just have to find it? or | 16:23 |
| *** jcoufal has joined #tripleo | 16:23 | |
| *** aditya_r has quit IRC | 16:23 | |
| mwhahaha | ansiwen: as in ec2api needs to implement one if one is currently not available | 16:23 |
| ansiwen | mwhahaha: or they are violating rules? | 16:23 |
| mwhahaha | ansiwen: no rules, it's a good practice | 16:23 |
| mwhahaha | for example glance has /healthcheck https://github.com/openstack/puppet-tripleo/blob/master/manifests/haproxy.pp#L1003 | 16:23 |
| ansiwen | mwhahaha: ok, I will try to find out if they have... | 16:24 |
| ansiwen | mwhahaha: is "healthcheck" the proper google keyword for that? or do you have another idea? | 16:24 |
| ansiwen | could also check the code I guess | 16:25 |
| mwhahaha | ansiwen: well a health check url is more of a concept | 16:25 |
| mwhahaha | the code is the best place to look | 16:25 |
| mwhahaha | you need a url that doesn't require auth for the service that is there to say i'm running | 16:25 |
| *** akrivoka has joined #tripleo | 16:26 | |
| mwhahaha | it's likely they didn't implement one so in the short term you'd need to switch to tcpcheck to ensure that the port is up | 16:26 |
| mwhahaha | but that doesn't actually verify the service is working correctly | 16:26 |
| *** achadha has quit IRC | 16:26 | |
| mwhahaha | this is just api service best practices | 16:26 |
| mwhahaha | ansiwen: btw the way to configure the tcp check rather than httpchk you'd need to use tcpka https://github.com/openstack/puppet-tripleo/blob/master/manifests/haproxy.pp#L1061 | 16:30 |
| ansiwen | mwhahaha: ec2-api is not rest api, but SOAP... maybe that's why they don't have that concept | 16:31 |
| ansiwen | mwhahaha: thanks... but "mode" could still be http, right? | 16:32 |
| *** dsariel__ has quit IRC | 16:35 | |
| *** hewbrocca is now known as hewbrocca_afk | 16:36 | |
| *** sri_ has quit IRC | 16:37 | |
| *** cylopez has quit IRC | 16:37 | |
| *** itlinux has quit IRC | 16:39 | |
| numans | EmilienM, Hi, can you please have a look at this small patch whenever you get time - https://review.openstack.org/#/c/502542/4. The concern from you earlier was that scenario007 (OVN) job was not run for this patch. It has now passed. | 16:39 |
| mwhahaha | ansiwen: yea | 16:40 |
| EmilienM | numans: sure | 16:41 |
| *** yamahata has joined #tripleo | 16:41 | |
| numans | thanks | 16:41 |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: [WIP] Add umount_safe function to get better debug info https://review.openstack.org/514081 | 16:43 |
| *** itlinux has joined #tripleo | 16:43 | |
| *** aufi has quit IRC | 16:43 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart-extras master: config-download: use workflow supported in Queens https://review.openstack.org/516841 | 16:48 |
| *** salmankhan has joined #tripleo | 16:48 | |
| *** marios has quit IRC | 16:48 | |
| *** dsariel__ has joined #tripleo | 16:48 | |
| *** ebarrera has quit IRC | 16:48 | |
| openstackgerrit | Sven Anderson proposed openstack/puppet-tripleo master: Change haproxy check to tcpka for ec2_api https://review.openstack.org/517655 | 16:50 |
| ansiwen | mwhahaha: ^^^ | 16:50 |
| *** tesseract has quit IRC | 16:54 | |
| *** links has quit IRC | 16:55 | |
| ansiwen | mwhahaha: thanks for your help! | 16:57 |
| *** jmelvin_ has quit IRC | 16:57 | |
| mwhahaha | ansiwen: i commented, do you know if you need the other default options for ssl or antyhing? | 16:59 |
| *** trown is now known as trown|lunch | 16:59 | |
| ansiwen | ssl is not there yet, it's implemented in another change | 16:59 |
| EmilienM | mwhahaha: fyi https://review.openstack.org/#/c/517022 can be merged once it pass, saw success on https://review.openstack.org/#/c/517440/2 | 16:59 |
| mwhahaha | orly | 17:00 |
| EmilienM | mwhahaha: yeah | 17:00 |
| ansiwen | mwhahaha: you think I should merge it with them in any case? | 17:00 |
| EmilienM | it was the biggest regression I've been lately | 17:00 |
| EmilienM | seen* | 17:00 |
| mwhahaha | ansiwen: yea | 17:00 |
| *** fragatina has joined #tripleo | 17:00 | |
| ansiwen | mwhahaha: ok, one sec... | 17:00 |
| *** fragatina has quit IRC | 17:01 | |
| *** fragatina has joined #tripleo | 17:01 | |
| *** chem has quit IRC | 17:02 | |
| openstackgerrit | Sven Anderson proposed openstack/puppet-tripleo master: Change haproxy check to tcpka for ec2_api https://review.openstack.org/517655 | 17:03 |
| ansiwen | mwhahaha: ^^^ | 17:04 |
| *** jcoufal has quit IRC | 17:05 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Run containerized mistral-api eventlet https://review.openstack.org/515326 | 17:05 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Fix bind mounts for cinder-{backup,volume} https://review.openstack.org/517221 | 17:05 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Make docker network configurable https://review.openstack.org/514576 | 17:05 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: create subunit test results for the undercloud sanity check https://review.openstack.org/517657 | 17:07 |
| *** brault has quit IRC | 17:07 | |
| *** brault has joined #tripleo | 17:08 | |
| *** ffiore has quit IRC | 17:09 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729881 | 17:10 |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 17:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 17:10 |
| openstack | Launchpad bug 1729881 in tripleo "401 unauthorized in overcloud-validate on ovb-containers job" [Critical,Triaged] | 17:10 |
| *** jtomasek has quit IRC | 17:12 | |
| *** aputtur_ has quit IRC | 17:12 | |
| *** brault has quit IRC | 17:12 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Upgrade CI test on stable/pike https://review.openstack.org/500625 | 17:13 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: create subunit test results for the undercloud sanity check https://review.openstack.org/517657 | 17:14 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: fix the undercloud-oooq job's subunit failures https://review.openstack.org/517428 | 17:15 |
| *** jprovazn has joined #tripleo | 17:16 | |
| *** jkilpatr has quit IRC | 17:16 | |
| openstackgerrit | K Jonathan Harker proposed openstack/diskimage-builder master: zypper: fix package removal https://review.openstack.org/517658 | 17:18 |
| *** aditya_ra has quit IRC | 17:19 | |
| *** mcornea has quit IRC | 17:19 | |
| *** yprokule has quit IRC | 17:20 | |
| *** moshele has joined #tripleo | 17:22 | |
| *** newmember has quit IRC | 17:23 | |
| *** jfrancoa has quit IRC | 17:23 | |
| *** ykarel|away has joined #tripleo | 17:24 | |
| *** jtomasek has joined #tripleo | 17:25 | |
| weshay | panda|ruck, fyi https://review.openstack.org/#/c/517428/ | 17:25 |
| *** lucasagomes is now known as lucas-afk | 17:27 | |
| *** jkilpatr has joined #tripleo | 17:28 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo stable/pike: Make docker network configurable https://review.openstack.org/517661 | 17:29 |
| *** salmankhan has quit IRC | 17:30 | |
| *** catintheroof has quit IRC | 17:31 | |
| *** catintheroof has joined #tripleo | 17:33 | |
| *** chlong_ has joined #tripleo | 17:34 | |
| beekneemech | EmilienM: mwhahaha weshay sshnaidm|off: FYI, I'm proposing https://review.openstack.org/517662 | 17:34 |
| mwhahaha | k | 17:34 |
| beekneemech | When we've been running at capacity today we're still seeing timeouts that don't seem to happen when the cloud is less loaded. | 17:34 |
| *** dtantsur is now known as dtantsur|afk | 17:38 | |
| *** ckyriakidou has quit IRC | 17:39 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart-extras master: config-download: use workflow supported in Queens https://review.openstack.org/516841 | 17:43 |
| *** noslzzp has joined #tripleo | 17:43 | |
| *** jprovazn has quit IRC | 17:44 | |
| *** catintheroof has quit IRC | 17:45 | |
| *** pblaho has quit IRC | 17:45 | |
| EmilienM | beekneemech: ack, soudns good | 17:45 |
| EmilienM | beekneemech: we could ask dmsimard to take a look at https://review.openstack.org/517662 | 17:46 |
| dmsimard | +2 | 17:47 |
| *** ykarel|away has quit IRC | 17:49 | |
| *** itlinux has quit IRC | 17:51 | |
| *** catintheroof has joined #tripleo | 17:53 | |
| *** dsneddon has joined #tripleo | 17:56 | |
| panda|ruck | weshay: every test will overwrite subunit file ? | 17:58 |
| weshay | panda|ruck, it exits on the first failure | 17:58 |
| *** jpich has quit IRC | 17:58 | |
| panda|ruck | weshay: se we are good not having the complete description of services that failed | 17:59 |
| *** itlinux has joined #tripleo | 18:01 | |
| weshay | panda|ruck, I think we'd see the command and stdout stderror in the log | 18:01 |
| panda|ruck | weshay: yes, but only for the first service that failed. If more than one fails, it can be inferred only looking at the logs. | 18:03 |
| panda|ruck | also we only have one failure in the subunit | 18:03 |
| *** derekh has quit IRC | 18:04 | |
| panda|ruck | weshay: it may be ok, but it's not what tempest does for example, doesn't stop at the first failure. Not the idea under a subunit file, I think. Anyway, I won't block that if we don't really care | 18:05 |
| panda|ruck | and we care only about the failure | 18:05 |
| weshay | panda|ruck, agree that is not what tempest does | 18:05 |
| *** jmelvin_ has joined #tripleo | 18:08 | |
| *** noslzzp has quit IRC | 18:09 | |
| *** trown|lunch is now known as trown | 18:09 | |
| *** salmankhan has joined #tripleo | 18:09 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729881 | 18:10 |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 18:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 18:10 |
| openstack | Launchpad bug 1729881 in tripleo "401 unauthorized in overcloud-validate on ovb-containers job" [Critical,Triaged] | 18:10 |
| *** catintheroof has quit IRC | 18:11 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common master: Updated from global requirements https://review.openstack.org/511633 | 18:12 |
| weshay | panda|ruck, https://trello.com/c/QLfk7H5y | 18:14 |
| *** catintheroof has joined #tripleo | 18:17 | |
| weshay | hrm.. openstack-tempest continues to bite us | 18:17 |
| weshay | http://logs.openstack.org/64/510464/23/gate/legacy-tripleo-ci-centos-7-scenario004-multinode-oooq-container/40b9d30/logs/undercloud/var/log/extra/yum-list-installed.txt.gz | 18:17 |
| weshay | ccamacho, fyi https://review.openstack.org/#/c/510464/ ^ | 18:18 |
| *** bogdando has quit IRC | 18:23 | |
| *** jpena is now known as jpena|off | 18:27 | |
| *** weshay is now known as weshay_brb | 18:30 | |
| *** brault has joined #tripleo | 18:30 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Certmonger: Only notify haproxy class if it's defined https://review.openstack.org/517023 | 18:32 |
| *** moshele has quit IRC | 18:33 | |
| *** brault has quit IRC | 18:35 | |
| openstackgerrit | Keith Schincke proposed openstack/puppet-tripleo master: Set ACLs on ceph client keyrings https://review.openstack.org/517347 | 18:38 |
| *** cylopez has joined #tripleo | 18:41 | |
| *** cylopez has quit IRC | 18:41 | |
| *** amoralej is now known as amoralej|off | 18:44 | |
| *** itlinux has quit IRC | 18:44 | |
| *** brault has joined #tripleo | 18:49 | |
| *** brault has quit IRC | 18:53 | |
| mwhahaha | weshay_brb: uh oh does that mean the gate's going to faill out? | 18:59 |
| mwhahaha | thought we fixed all the tempest stuff :/ | 19:00 |
| mwhahaha | http://logs.openstack.org/64/510464/23/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/eec1b49/job-output.txt.gz#_2017-11-03_18_14_56_665149 | 19:00 |
| *** brault has joined #tripleo | 19:01 | |
| *** brault has quit IRC | 19:05 | |
| tonyb | Thanks for the help yesterday I got the containers running but I still have the isses that whatever is supposed to manage the VIPs (haproxy?) isn't so nothing can talk to the services (specifially mysql). | 19:07 |
| tonyb | I've looked through the os-collect-config logs and I don't see anything that seems to indicate an error? | 19:08 |
| *** salmankhan has quit IRC | 19:08 | |
| tonyb | any ideas? | 19:08 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729881 | 19:10 |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 19:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 19:10 |
| openstack | Launchpad bug 1729881 in tripleo "401 unauthorized in overcloud-validate on ovb-containers job" [Critical,Triaged] | 19:10 |
| *** apevec has joined #tripleo | 19:14 | |
| mwhahaha | tonyb: yea it's haproxy | 19:14 |
| *** itlinux has joined #tripleo | 19:14 | |
| mwhahaha | tonyb: are the ports listening on the controller? or is that not even started | 19:15 |
| tonyb | mwhahaha: haproxy seems to be running as a process and inside a container | 19:15 |
| tonyb | mwhahaha: which ports does haproxy use? | 19:16 |
| mwhahaha | haproxy is the one that'll listen on all the ports | 19:16 |
| mwhahaha | for the vip | 19:16 |
| mwhahaha | so is it failing cause that's not working or is it failing because the backend is down | 19:16 |
| mwhahaha | like if you telnet to 3306 on the vip, is it answering or connection refused | 19:16 |
| tonyb | netstat -tpan | grep haproxy has a bunch od likley looking ports | 19:17 |
| mwhahaha | so haproxy is probably working | 19:18 |
| mwhahaha | what might be down is the backend servers | 19:18 |
| *** rwsu has quit IRC | 19:18 | |
| mwhahaha | there's a status port/page thingy that you can query for haproxy to tell you the state | 19:18 |
| *** salmankhan has joined #tripleo | 19:18 | |
| mwhahaha | tonyb: i've always used to https://github.com/openstack/fuel-library/blob/master/files/fuel-misc/haproxy-status.sh (after modifying the connection info) | 19:19 |
| mwhahaha | so you could try that in the haproxy container to see what's up | 19:19 |
| tonyb | mwhahaha: So the controller has 192.168.0.78 and haproxy is listening in in .72:<lots> | 19:19 |
| mwhahaha | i've not tried it with containerized haproxy lately | 19:19 |
| mwhahaha | so like if the mysql backend is down in haproxy, the health checks are failing so we would need to go to the next step of troubleshooting | 19:19 |
| tonyb | mwhahaha: but I get 'Ncat: No route to host.' so it seems like something failed to create the actual vip on an interface? | 19:20 |
| mwhahaha | haproxy would be listening but would lead to connection failures in the logs | 19:20 |
| *** weshay_brb is now known as weshay | 19:20 | |
| mwhahaha | not sure i think it's just a bridge, is this HA or nonha | 19:20 |
| weshay | mwhahaha, the jobs I've seen failing on tempest are still using old rpms | 19:21 |
| mwhahaha | weshay: :( | 19:21 |
| weshay | mwhahaha, if they started 20 hrs ago | 19:21 |
| weshay | I guess that's what happens | 19:21 |
| mwhahaha | weshay: no couldn't be | 19:21 |
| mwhahaha | they were reset like <2 hours ago | 19:21 |
| tonyb | mwhahaha: I have tried passing both docker-ha.yaml and not I'm pretty sure this run is with it | 19:21 |
| tonyb | mwhahaha: let me confirm | 19:21 |
| *** brault has joined #tripleo | 19:22 | |
| tonyb | mwhahaha: Oh it's without. | 19:22 |
| mwhahaha | so pacemaker wouldn't be there, that's what we test in CI so it should be working | 19:23 |
| mwhahaha | something else must be misconfigured | 19:23 |
| weshay | http://logs.openstack.org/64/510464/23/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/eec1b49/logs/undercloud/etc/yum.repos.d/delorean.repo.txt.gz | 19:23 |
| * weshay looks around for why | 19:23 | |
| weshay | != https://trunk.rdoproject.org/centos7-master/current-tripleo/delorean.repo | 19:24 |
| openstack | weshay: Error: "=" is not a valid command. | 19:24 |
| tonyb | mwhahaha: pacemaker (pcs) is running but doesn't have any resource (according to pcs status) | 19:24 |
| mwhahaha | tonyb: yea without docker-ha.yaml i don't think it configures it | 19:24 |
| *** jrist has quit IRC | 19:25 | |
| tonyb | mwhahaha: I looked at the haproxy container logs and it's basically 'starting' | 19:25 |
| tonyb | mwhahaha: okay | 19:25 |
| apevec | weshay, is it b/c of upstream proxy cache? | 19:25 |
| weshay | maybe.. but it looks like it's there http://mirror.dfw.rax.openstack.org:8080/rdo/centos7/tripleo-ci-testing/delorean.repo | 19:26 |
| weshay | 6c/d7/6cd7161a7f785abd817e1743f60341acebbdb697_ae492027 | 19:26 |
| mwhahaha | maybe? | 19:26 |
| weshay | oh that is the issue | 19:26 |
| *** brault has quit IRC | 19:26 | |
| apevec | you mean http://mirror.dfw.rax.openstack.org:8080/rdo/centos7/current-tripleo/delorean.repo ? | 19:27 |
| mwhahaha | the proxy request for the current-tripleo file? | 19:27 |
| tonyb | mwhahaha: so I can try the deploy again and manually add the IP to the controller and see if that helps but it's be nice to know what's failing to do that for me | 19:27 |
| weshay | https://trunk.rdoproject.org/centos7-master/current-tripleo/delorean.repo is at 3b/71/3b718f3fecc866332ec0663fa77e758f8346ab93_4204ba89 | 19:27 |
| mwhahaha | tonyb: you shouldn't need to do that | 19:27 |
| mwhahaha | tonyb: sounds like the network config is bad or some other configuration | 19:27 |
| apevec | weshay, yeah, so it is still old cached... | 19:28 |
| mwhahaha | weshay: it might be now, but it might not have been 2 hours ago | 19:28 |
| tonyb | mwhahaha: okay. I'm pretty sure I've messed this up but I'm kinda stumped as to what or how ;P | 19:28 |
| weshay | maybe it's just one of the clouds | 19:28 |
| mwhahaha | tonyb: i can has templates? | 19:28 |
| weshay | providers | 19:28 |
| apevec | pabelanger, what is the cache expiry for upstream proxy caches? | 19:28 |
| tonyb | mwhahaha: Sure | 19:29 |
| apevec | weshay, shift-reload on http://mirror.dfw.rax.openstack.org:8080/rdo/centos7/current-tripleo/delorean.repo got me new .repo ... | 19:29 |
| tonyb | mwhahaha: how do you want them. I can get you onto the director it that works or I can tarball up whatever you need | 19:29 |
| mwhahaha | tonyb: on the director would probably be easier | 19:30 |
| tonyb | mwhahaha: okay | 19:30 |
| weshay | heh.. thanks | 19:30 |
| mwhahaha | weshay: wonder if we need to add some sort of cache buster for the .repo files | 19:31 |
| tonyb | mwhahaha: let me setup a tunnel on my jump host | 19:31 |
| weshay | mwhahaha, ya.. to force an expire on promotion | 19:31 |
| weshay | pabelanger, is that possible? | 19:32 |
| mwhahaha | weshay: or i wonder if adding like ?id=<timestamp> will bypass cache | 19:32 |
| *** pcaruana has quit IRC | 19:32 | |
| mwhahaha | sometimes it will | 19:32 |
| pabelanger | apevec: we expire cache every 24 hours with htcacheclean, however we don't do anything with HTTP headers | 19:32 |
| apevec | weshay, but you would need to do it across all providers? | 19:32 |
| weshay | ya | 19:32 |
| pabelanger | well, you can configure apache upstream (RDO) for different headers expire | 19:33 |
| pabelanger | and our apache2 reverse proxy cache will respect them | 19:33 |
| pabelanger | the only one we don't is hub.docker.com, we ignore expires from them, since they hash with sha1, they'll never change | 19:34 |
| pabelanger | apevec: weshay: does that help? | 19:35 |
| weshay | I'd have to chat w/ apevec and dmsimard | 19:35 |
| *** jrist has joined #tripleo | 19:36 | |
| pabelanger | yah, basically our reverse proxy is transparent, any changes to caching should be done at the source, in this case RDO | 19:36 |
| pabelanger | and we'll respect them | 19:36 |
| mwhahaha | maybe a filesmatch expiresbytype config? | 19:36 |
| mwhahaha | or we could update quickstart to append a query string that might bypass the cache | 19:36 |
| mwhahaha | ie request repo files with ?id=timestamp | 19:36 |
| pabelanger | yah, seem hacky. just lower it on RDO apache to something sane, 60mins | 19:37 |
| mwhahaha | well we don't want any | 19:37 |
| mwhahaha | for the repo files | 19:37 |
| mwhahaha | cause promotion | 19:37 |
| *** jtomasek has quit IRC | 19:38 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-common master: WIP add UC backup actions https://review.openstack.org/517610 | 19:38 |
| apevec | we have something WIP https://review.rdoproject.org/r/10345 for repomd.xml | 19:39 |
| pabelanger | mwhahaha: sure, they just setup RDO apache to disable caching on that file | 19:39 |
| apevec | please comment there | 19:39 |
| pabelanger | there was a patch to do it for repomd | 19:39 |
| pabelanger | ah, the change apevec linked | 19:39 |
| apevec | yeah | 19:39 |
| apevec | but w/ caching disabled we have spof on trunk.rdo ? | 19:40 |
| pabelanger | I wouldn't disable caching, I'd set it to something like 60mins | 19:40 |
| apevec | ok | 19:40 |
| weshay | panda|ruck, hrm.. pike should have promoted in tripleo, but I don't see it | 19:40 |
| mwhahaha | well it can lead to incosistent runs | 19:41 |
| apevec | b/c current WIP has Pragma "no-cache"', 'set Cache-Control "max-age=0, no-cache, no-store, must-revalidate | 19:41 |
| pabelanger | otherwise, jobs are going to start hammering RDO again, and could result in more failures due to network pressure | 19:41 |
| pabelanger | mwhahaha: yah, my concern too | 19:41 |
| apevec | or jpena's inline comment has... | 19:41 |
| weshay | panda|ruck, https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-pike-upload/236/ | 19:42 |
| apevec | weshay, all -pike are success | 19:42 |
| apevec | master were abort b/c it was before pin revert was merged | 19:42 |
| weshay | ya | 19:42 |
| weshay | ya | 19:42 |
| weshay | oh I see why | 19:44 |
| weshay | missing successful jobs: ['periodic-multinode-1ctlr-featureset016', 'periodic-multinode-1ctlr-featureset019'] | 19:44 |
| weshay | there it goes | 19:46 |
| weshay | promoter SUCCESS promoting tripleo-ci-testing as current-tripleo (old: {'commit_hash': 'd558d6173970736fde63526da6824d391ada6042', 'distro_hash': '89d45961e545c289c4b8d6c2a0dfd63d909c0327'}, new: {'full_hash': '45933d59349c67a32fe09c3192e26fa57e226270_f7079069', 'commit_hash': '45933d59349c67a32fe09c3192e26fa57e226270', 'distro_hash': 'f7079069eb875af50c0df58d8ebb50cbb023c2b8'}) | 19:46 |
| weshay | 2017-11-03 19:42:15,191 18182 INFO promoter Promoting the container images for dlrn hash 45933d59349c67a32fe09c3192e26fa57e226270 on pike to current-tripleo | 19:46 |
| apevec | weshay, but they are reported as SUCCESS in Jenkins?! | 19:46 |
| weshay | apevec, I guess it just passed in the last 10 minutes | 19:47 |
| apevec | ah | 19:47 |
| weshay | we only check every 10 minutes | 19:47 |
| weshay | https://trunk.rdoproject.org/centos7-pike/current-tripleo/ looks good | 19:47 |
| *** jrist_ has joined #tripleo | 19:49 | |
| apevec | weshay, what is the frequency of Jenkins URLTrigger ? | 19:49 |
| *** jrist_ has quit IRC | 19:49 | |
| *** jrist has quit IRC | 19:49 | |
| weshay | apevec, for which step? | 19:50 |
| apevec | https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-pike-current-tripleo/urltriggerPollLog/ | 19:50 |
| apevec | rdo phase1 | 19:50 |
| *** jrist has joined #tripleo | 19:50 | |
| *** jrist has joined #tripleo | 19:50 | |
| weshay | apevec, I'll check, but don't kick it yet please.. the images and containers are still transfering | 19:51 |
| weshay | https://images.rdoproject.org/pike/rdo_trunk/current-tripleo/ | 19:51 |
| weshay | apevec, https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/promote-pike-current-tripleo.yml#L41 | 19:53 |
| *** dparkes has joined #tripleo | 19:54 | |
| *** panda|ruck is now known as panda|ruck|off | 19:56 | |
| weshay | trown, ping, need your eyes for a hot sec | 19:57 |
| Slower | weshay: did you notice this? | 19:59 |
| Slower | http://logs.openstack.org/44/517444/1/check/legacy-tripleo-ci-centos-7-containers-multinode/2010544/logs/undercloud/home/zuul/undercloud-install.sh.txt.gz | 19:59 |
| weshay | ah no, haven't had a chance to look at that review at all | 20:00 |
| weshay | Slower, guess it just needs to be blank | 20:00 |
| *** achadha has joined #tripleo | 20:00 | |
| Slower | weshay: ok I can try to fix it.. | 20:00 |
| *** achadha has quit IRC | 20:01 | |
| *** achadha has joined #tripleo | 20:01 | |
| *** catintheroof has quit IRC | 20:04 | |
| *** achadha_ has quit IRC | 20:04 | |
| *** dougbtv_ has quit IRC | 20:05 | |
| *** achadha has quit IRC | 20:06 | |
| *** rfolco has quit IRC | 20:06 | |
| openstackgerrit | Ian Main proposed openstack/tripleo-quickstart-extras master: DNM, update the undercloud install for containers https://review.openstack.org/517444 | 20:08 |
| weshay | dmsimard, is the rdo registry restarting? | 20:09 |
| dmsimard | weshay: I'm not doing anything on it right now but apevec mentioned an update was perhaps in order | 20:09 |
| weshay | k | 20:10 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729881 | 20:10 |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 20:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 20:10 |
| openstack | Launchpad bug 1729881 in tripleo "401 unauthorized in overcloud-validate on ovb-containers job" [Critical,Triaged] | 20:10 |
| *** rwsu has joined #tripleo | 20:14 | |
| *** catintheroof has joined #tripleo | 20:15 | |
| *** eck` is now known as eck`gone | 20:17 | |
| *** ansmith has quit IRC | 20:19 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Additional tests for config-download https://review.openstack.org/511506 | 20:22 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Skip hidden files in test_tags_are_set https://review.openstack.org/512875 | 20:22 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Add workflow for deployment via config-download https://review.openstack.org/512876 | 20:22 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Log Ansible actions to ansible.log https://review.openstack.org/514783 | 20:22 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Add work_dir for some actions https://review.openstack.org/514784 | 20:22 |
| openstackgerrit | Ian Main proposed openstack/tripleo-quickstart master: DNM, update undercloud install options for containers https://review.openstack.org/517445 | 20:22 |
| *** dparkes has quit IRC | 20:24 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Upgrade rhel_reg_sat_repo to 6.2 https://review.openstack.org/517138 | 20:25 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: mysql: Only set certificate specs if TLS everywhere is enabled https://review.openstack.org/517029 | 20:25 |
| *** achadha has joined #tripleo | 20:25 | |
| trown | weshay: sure whats up? | 20:25 |
| *** achadha_ has joined #tripleo | 20:27 | |
| *** achadha has quit IRC | 20:30 | |
| *** dparkes has joined #tripleo | 20:30 | |
| *** achadha has joined #tripleo | 20:30 | |
| *** itlinux has quit IRC | 20:31 | |
| *** achadha_ has quit IRC | 20:31 | |
| *** achadha has quit IRC | 20:35 | |
| *** itlinux has joined #tripleo | 20:37 | |
| mwhahaha | meh more failures due to the cached .repo file | 20:38 |
| apevec | weshay, mwhahaha, ^ manual workaround would be to shift-reload on all mirrors, do we have list? | 20:39 |
| mwhahaha | yea would have to get one together | 20:40 |
| *** brault has joined #tripleo | 20:48 | |
| *** trown is now known as trown|outtypewww | 20:51 | |
| *** brault has quit IRC | 20:52 | |
| *** salmankhan has quit IRC | 20:55 | |
| tonyb | EmilienM: FWIW a spot checked pike/paunch and it's on tarballs.o.o and pypi | 20:59 |
| EmilienM | yea | 20:59 |
| EmilienM | checked tripleo-ui, tht and puppet-tripleo | 20:59 |
| EmilienM | all good | 20:59 |
| EmilienM | best friday ever | 21:00 |
| *** bswartz has quit IRC | 21:02 | |
| *** abishop has quit IRC | 21:03 | |
| *** catintheroof has quit IRC | 21:04 | |
| apevec | EmilienM, take a snapshot! https://dashboards.rdoproject.org/rdo-dev | 21:05 |
| EmilienM | apevec: this is fake HTML. | 21:07 |
| *** SlickNik has left #tripleo | 21:08 | |
| EmilienM | apevec: or is it by bday present? :P | 21:08 |
| *** SlickNik has joined #tripleo | 21:09 | |
| *** SlickNik has left #tripleo | 21:09 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729881 | 21:10 |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 21:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 21:10 |
| openstack | Launchpad bug 1729881 in tripleo "401 unauthorized in overcloud-validate on ovb-containers job" [Critical,Triaged] | 21:10 |
| *** raildo has quit IRC | 21:11 | |
| *** ansmith has joined #tripleo | 21:15 | |
| *** SlickNik has joined #tripleo | 21:25 | |
| *** mary_grace has joined #tripleo | 21:26 | |
| *** achadha has joined #tripleo | 21:26 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-repos master: Update ceph and opstools repos https://review.openstack.org/517728 | 21:30 |
| *** achadha has quit IRC | 21:31 | |
| *** itlinux has quit IRC | 21:49 | |
| *** paramite has quit IRC | 21:49 | |
| *** apevec has left #tripleo | 21:51 | |
| *** jmelvin_ has quit IRC | 21:52 | |
| *** d0ugal has quit IRC | 22:06 | |
| *** fragatina has quit IRC | 22:06 | |
| *** d0ugal has joined #tripleo | 22:10 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729881 | 22:10 |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 22:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 22:10 |
| openstack | Launchpad bug 1729881 in tripleo "401 unauthorized in overcloud-validate on ovb-containers job" [Critical,Triaged] | 22:10 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Upgrade CI test on stable/pike https://review.openstack.org/500625 | 22:10 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Upgrade CI test on stable/pike https://review.openstack.org/500625 | 22:11 |
| *** jrist has quit IRC | 22:11 | |
| *** jrist has joined #tripleo | 22:15 | |
| *** jrist has joined #tripleo | 22:15 | |
| *** thrash is now known as thrash|g0ne | 22:24 | |
| *** achadha has joined #tripleo | 22:38 | |
| *** achadha has quit IRC | 22:42 | |
| *** bobh has quit IRC | 22:47 | |
| *** itlinux has joined #tripleo | 22:50 | |
| *** fragatina has joined #tripleo | 22:50 | |
| *** ckyriakidou has joined #tripleo | 22:51 | |
| *** mary_grace has quit IRC | 22:53 | |
| *** itlinux has quit IRC | 22:53 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: create subunit test results for the undercloud sanity check https://review.openstack.org/517657 | 22:53 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Fix nova-cpu/collectd hieradata https://review.openstack.org/517352 | 22:57 |
| *** adac has quit IRC | 23:00 | |
| *** itlinux has joined #tripleo | 23:01 | |
| *** itlinux has quit IRC | 23:04 | |
| *** ckyriakidou has quit IRC | 23:07 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729253 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729328 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729586 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1729881 | 23:10 |
| openstack | Launchpad bug 1729253 in tripleo "CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found" [Critical,In progress] - Assigned to Martin André (mandre) | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 23:10 |
| openstack | Launchpad bug 1729586 in tripleo "CI: rdocloud node randomly going offline during jobs" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 23:10 |
| openstack | Launchpad bug 1729881 in tripleo "401 unauthorized in overcloud-validate on ovb-containers job" [Critical,Triaged] | 23:10 |
| *** toure is now known as toure_biab | 23:15 | |
| openstackgerrit | Keith Schincke proposed openstack/puppet-tripleo master: Set ACLs on ceph client keyrings https://review.openstack.org/517347 | 23:23 |
| *** links has joined #tripleo | 23:32 | |
| *** lblanchard has quit IRC | 23:43 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!