| *** mwhahaha changes topic to "CI status: ORANGE (zuul restart lost jobs, scenario001 broken in master) | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest/" | 00:00 | |
| *** dsneddon has quit IRC | 00:05 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 00:10 |
| * mwhahaha has loaded up CI enough for the day | 00:10 | |
| mwhahaha | rechecks for everyone! WOOO | 00:10 |
| *** ebarrera has quit IRC | 00:10 | |
| EmilienM | mwhahaha: why do you think I did https://review.openstack.org/#/c/529872/ | 00:13 |
| EmilienM | sshnaidm|afk, weshay : look at that https://review.openstack.org/#/c/526481/ | 00:21 |
| EmilienM | the jobs take longer now | 00:21 |
| *** moshele has joined #tripleo | 00:21 | |
| EmilienM | I wonder if it's because of Meltdown etc | 00:21 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Containerize multinode-3nodes https://review.openstack.org/532623 | 00:23 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Add DockerPuppetDebug to environments/config-debug.yaml https://review.openstack.org/532728 | 00:24 |
| *** moshele has quit IRC | 00:26 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo stable/newton: Create dedicated "apache" base profile https://review.openstack.org/532742 | 00:30 |
| *** jobewan has quit IRC | 00:34 | |
| *** bfournie has joined #tripleo | 00:35 | |
| *** bfournie has quit IRC | 00:37 | |
| *** bfournie has joined #tripleo | 00:43 | |
| *** tosky has quit IRC | 00:51 | |
| *** lblanchard has joined #tripleo | 00:55 | |
| *** threestrands has quit IRC | 00:58 | |
| *** itlinux has joined #tripleo | 01:01 | |
| *** threestrands has joined #tripleo | 01:02 | |
| *** etingof has quit IRC | 01:03 | |
| *** threestrands has quit IRC | 01:03 | |
| *** threestrands has joined #tripleo | 01:04 | |
| *** bfournie has left #tripleo | 01:07 | |
| *** akrivoka has quit IRC | 01:08 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 01:10 |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| *** mdnadeem has joined #tripleo | 01:10 | |
| *** threestrands has quit IRC | 01:11 | |
| *** jlabarre has quit IRC | 01:13 | |
| *** hamzy has joined #tripleo | 01:17 | |
| *** dpawar has joined #tripleo | 01:19 | |
| *** dpawar has quit IRC | 01:20 | |
| *** dpawar has joined #tripleo | 01:20 | |
| *** numans has quit IRC | 01:24 | |
| *** threestrands has joined #tripleo | 01:24 | |
| *** numans has joined #tripleo | 01:25 | |
| *** threestrands has quit IRC | 01:25 | |
| *** threestrands has joined #tripleo | 01:26 | |
| *** numans has quit IRC | 01:29 | |
| *** numans has joined #tripleo | 01:35 | |
| *** ebarrera has joined #tripleo | 01:36 | |
| *** dhill__ has quit IRC | 01:41 | |
| *** etingof has joined #tripleo | 01:43 | |
| *** liverpooler has joined #tripleo | 01:49 | |
| *** dpawar has quit IRC | 01:53 | |
| *** threestrands has quit IRC | 01:54 | |
| *** threestrands has joined #tripleo | 01:56 | |
| *** bfournie has joined #tripleo | 02:01 | |
| *** ramishra has joined #tripleo | 02:08 | |
| *** yamahata has quit IRC | 02:09 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 02:10 |
| *** thrash is now known as thrash|g0ne | 02:11 | |
| openstackgerrit | Merged openstack/instack-undercloud stable/ocata: Set password for mysql root user on undercloud https://review.openstack.org/532806 | 02:11 |
| openstackgerrit | Ian Main proposed openstack/tripleo-quickstart-extras master: DNM: Update quickstart extras with undercloud install for containers https://review.openstack.org/517444 | 02:13 |
| *** kambiz_ has quit IRC | 02:13 | |
| *** pmannidi has joined #tripleo | 02:15 | |
| *** kambiz has joined #tripleo | 02:16 | |
| *** liverpooler has quit IRC | 02:16 | |
| *** gkadam has joined #tripleo | 02:18 | |
| *** ramishra has quit IRC | 02:27 | |
| *** dhill__ has joined #tripleo | 02:33 | |
| *** gkadam has quit IRC | 02:42 | |
| *** aputtur_ has joined #tripleo | 02:54 | |
| *** ecerquei has joined #tripleo | 02:55 | |
| openstackgerrit | Merged openstack/instack-undercloud stable/pike: Set password for mysql root user on undercloud https://review.openstack.org/532805 | 02:57 |
| *** yamahata has joined #tripleo | 03:02 | |
| *** numans has quit IRC | 03:05 | |
| *** numans has joined #tripleo | 03:07 | |
| *** ecerquei has quit IRC | 03:09 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 03:10 |
| *** lblanchard has quit IRC | 03:17 | |
| *** itlinux has quit IRC | 03:19 | |
| *** dhill__ has quit IRC | 03:24 | |
| *** aputtur_ has quit IRC | 03:35 | |
| *** fragatina has quit IRC | 03:37 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: zuul: run scenarios when touching pacemaker services https://review.openstack.org/533054 | 03:37 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo stable/pike: zuul: run scenarios when touching pacemaker services https://review.openstack.org/533057 | 03:38 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo stable/pike: cinder/pacemaker: resolve puppet resource duplications https://review.openstack.org/533058 | 03:40 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo stable/pike: Add profile to configure the rsyslog sidecar container https://review.openstack.org/533060 | 03:44 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Add rsyslog-sidecar resource and configuration https://review.openstack.org/533061 | 03:48 |
| *** larsks has quit IRC | 03:56 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Use docker_config_scripts for puppet apply https://review.openstack.org/533062 | 03:56 |
| *** larsks has joined #tripleo | 03:56 | |
| *** pdeore has joined #tripleo | 04:06 | |
| *** shreshtha has joined #tripleo | 04:07 | |
| *** shreshtha has quit IRC | 04:09 | |
| *** shreshtha has joined #tripleo | 04:10 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 04:10 |
| *** tzumainn has joined #tripleo | 04:18 | |
| EmilienM | bandini, dciabrin: when you're up, maybe you can help me to figure out why when I enable containers/pacemaker on multinode-3nodes, it timeouts badly http://logs.openstack.org/24/532624/4/check/tripleo-ci-centos-7-3nodes-multinode/30ccbad/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-01-12_03_58_41 | 04:20 |
| EmilienM | bandini, dciabrin: from the logs it seems like mysql isn't up correctly http://logs.openstack.org/24/532624/4/check/tripleo-ci-centos-7-3nodes-multinode/30ccbad/logs/subnode-2/var/log/extra/docker/containers/heat_engine_db_sync/stdout.log.txt.gz#_2018-01-12_03_10_39_066 | 04:23 |
| *** links has joined #tripleo | 04:31 | |
| *** itlinux has joined #tripleo | 04:32 | |
| *** pgadiya has joined #tripleo | 04:34 | |
| *** udesale has joined #tripleo | 04:37 | |
| *** pmannidi has quit IRC | 04:38 | |
| *** tzumainn has quit IRC | 04:40 | |
| *** anilvenkata has joined #tripleo | 04:41 | |
| EmilienM | Jan 12 03:09:45 centos-7-rax-ord-0001833276 haproxy[15611]: Backup Server mysql/centos-7-rax-ord-0001833276.internalapi.localdomain is UP, reason: Layer4 check passed, check duration: 0ms. 0 active and 1 backup servers online. Running on backup. 0 sessions requeued, 0 total in queue. | 04:42 |
| EmilienM | I want to know why mysql starts in backup mode | 04:42 |
| EmilienM | also http://logs.openstack.org/24/532624/4/check/tripleo-ci-centos-7-3nodes-multinode/30ccbad/logs/subnode-2/var/log/extra/docker/containers/heat_engine_db_sync/stdout.log.txt.gz#_2018-01-12_03_11_44_157 makes me thing at a networking issue | 04:44 |
| *** masco has joined #tripleo | 04:44 | |
| *** psachin has joined #tripleo | 04:52 | |
| openstackgerrit | Ian Main proposed openstack/tripleo-quickstart-extras master: DNM: Update quickstart extras with undercloud install for containers https://review.openstack.org/517444 | 04:52 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Containerize multinode-3nodes https://review.openstack.org/532623 | 04:56 |
| *** pmannidi has joined #tripleo | 05:00 | |
| itlinux | hello guys. I wonder if anyone has pushed mistral with OOO .. I remember there was an issue within ocata.. is it fixed in Pike? Thanks | 05:02 |
| EmilienM | bug 1731032 | 05:07 |
| openstack | bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] https://launchpad.net/bugs/1731032 | 05:07 |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 05:10 |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| *** dpawar has joined #tripleo | 05:19 | |
| *** ykarel|away has joined #tripleo | 05:22 | |
| *** absubram has joined #tripleo | 05:23 | |
| *** absubram has quit IRC | 05:23 | |
| *** ratailor has joined #tripleo | 05:24 | |
| *** akane_ has joined #tripleo | 05:24 | |
| Tengu | hello there | 05:31 |
| EmilienM | and... I'm gone :P | 05:31 |
| bandini | morning | 05:34 |
| *** gkadam has joined #tripleo | 05:34 | |
| EmilienM | bandini: hey | 05:38 |
| bandini | EmilienM: I see you spotted the problem already? | 05:39 |
| EmilienM | bandini: I'm about to leave to go sleep now, but I've dropped some notes ^ | 05:39 |
| bandini | EmilienM: I'll chase that review (+backport) | 05:39 |
| EmilienM | bandini: yeah but I haven't really found anything useful | 05:39 |
| EmilienM | bandini: I know heat-dbsync can't talk to MySQL which is on another node | 05:39 |
| bandini | oh so we found an error but not the reason for the timeout | 05:39 |
| bandini | got it | 05:39 |
| EmilienM | bandini: which problem are you talking about btw? | 05:40 |
| bandini | I'll spend some time on that | 05:40 |
| EmilienM | I'm talking about "/me trying to enable pacemaker/docker on the multinode-3nodes job, which is non voting" | 05:40 |
| EmilienM | bandini: if you're talking about https://bugs.launchpad.net/tripleo/+bug/1742795 - it's handled with https://review.openstack.org/#/c/532986/ (thx for +A btw) | 05:40 |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 05:40 |
| EmilienM | anyway, I'm off now, have a good day ;-) | 05:41 |
| bandini | EmilienM: I see. Where do I get logs for the timout 3nodes job issue? | 05:41 |
| bandini | ack I'll find them ;) | 05:41 |
| EmilienM | http://logs.openstack.org/24/532624/4/check/tripleo-ci-centos-7-3nodes-multinode/30ccbad/logs/ | 05:41 |
| bandini | got it! have a good night/evening ;) | 05:41 |
| EmilienM | thank you for the help :D | 05:41 |
| * EmilienM disappears | 05:41 | |
| *** moshele has joined #tripleo | 05:43 | |
| *** akane_ has quit IRC | 05:46 | |
| *** aputtur has quit IRC | 05:54 | |
| *** aputtur has joined #tripleo | 05:59 | |
| *** fragatina has joined #tripleo | 06:00 | |
| *** moshele has quit IRC | 06:02 | |
| *** fragatina has quit IRC | 06:04 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 06:10 |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| *** ykarel|away is now known as ykarel | 06:17 | |
| *** jfrancoa has joined #tripleo | 06:25 | |
| *** marios has joined #tripleo | 06:25 | |
| *** absubram has joined #tripleo | 06:26 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui stable/ocata: Imported Translations from Zanata https://review.openstack.org/533086 | 06:28 |
| *** suuuper has joined #tripleo | 06:29 | |
| *** dbecker has quit IRC | 06:31 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui stable/pike: Imported Translations from Zanata https://review.openstack.org/533088 | 06:31 |
| *** bkopilov has quit IRC | 06:40 | |
| *** dbecker has joined #tripleo | 06:44 | |
| *** gkadam has quit IRC | 06:45 | |
| *** pmannidi has quit IRC | 06:51 | |
| *** pdeore has quit IRC | 06:53 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-quickstart master: Containerize featureset032 (multinode-3nodes) https://review.openstack.org/532624 | 06:54 |
| *** pdeore has joined #tripleo | 06:54 | |
| *** jbadiapa has joined #tripleo | 06:56 | |
| *** daidv has quit IRC | 06:57 | |
| *** daidv has joined #tripleo | 06:58 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Start using ipv6 nic-configs from THT repo instead of tripleo-ci https://review.openstack.org/532235 | 07:00 |
| *** tdasilva has quit IRC | 07:07 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 07:10 |
| *** moshele has joined #tripleo | 07:10 | |
| *** aufi has joined #tripleo | 07:12 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Start using nic-configs from THT repo instead of tripleo-ci https://review.openstack.org/532236 | 07:14 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Add PublicVirtualFixedIPs to ci network-environment.yaml https://review.openstack.org/532240 | 07:14 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Use relative path in ipv6 nic configs https://review.openstack.org/532241 | 07:14 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Enhance completion message when upgrading non controller nodes https://review.openstack.org/482868 | 07:14 |
| *** itlinux has quit IRC | 07:20 | |
| *** dparkes has joined #tripleo | 07:20 | |
| *** tdasilva has joined #tripleo | 07:21 | |
| *** rcernin has quit IRC | 07:28 | |
| *** pcaruana has joined #tripleo | 07:29 | |
| *** amoralej|off is now known as amoralej | 07:38 | |
| *** yprokule has joined #tripleo | 07:39 | |
| *** rlandy|bbl has quit IRC | 07:44 | |
| *** threestrands has quit IRC | 07:56 | |
| *** tosky has joined #tripleo | 07:56 | |
| *** nyechiel has joined #tripleo | 08:03 | |
| *** cylopez has joined #tripleo | 08:05 | |
| *** tesseract has joined #tripleo | 08:08 | |
| *** florianf has quit IRC | 08:08 | |
| *** cylopez has left #tripleo | 08:08 | |
| *** florianf has joined #tripleo | 08:09 | |
| *** fzdarsky has joined #tripleo | 08:09 | |
| *** gkadam has joined #tripleo | 08:10 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 08:10 |
| *** moshele has quit IRC | 08:13 | |
| *** jtomasek has joined #tripleo | 08:15 | |
| *** pchavva has joined #tripleo | 08:21 | |
| *** shardy has joined #tripleo | 08:24 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/diskimage-builder master: provide names for arm64 grub packages https://review.openstack.org/533126 | 08:24 |
| *** pchavva has quit IRC | 08:26 | |
| *** cshastri has joined #tripleo | 08:30 | |
| *** ramishra has joined #tripleo | 08:32 | |
| *** ccamacho has joined #tripleo | 08:37 | |
| *** udesale has quit IRC | 08:37 | |
| *** ccamacho has joined #tripleo | 08:37 | |
| *** aditya_r has joined #tripleo | 08:38 | |
| *** holser__ has joined #tripleo | 08:44 | |
| *** lvdombrkr has joined #tripleo | 08:44 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: Use network templates from tripleo-heat-templates https://review.openstack.org/531221 | 08:45 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test configs on rh1 https://review.openstack.org/533129 | 08:47 |
| *** masco has quit IRC | 08:50 | |
| *** jpena|off is now known as jpena | 08:51 | |
| *** aufi_ has joined #tripleo | 08:52 | |
| *** aufi_ has quit IRC | 08:54 | |
| *** aufi has quit IRC | 08:54 | |
| *** aufi_ has joined #tripleo | 08:54 | |
| *** jpich has joined #tripleo | 08:55 | |
| *** bogdando has joined #tripleo | 09:00 | |
| *** ramishra has quit IRC | 09:02 | |
| *** masco has joined #tripleo | 09:04 | |
| *** dsneddon has joined #tripleo | 09:04 | |
| *** dsneddon has quit IRC | 09:05 | |
| *** dsneddon has joined #tripleo | 09:07 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 09:10 |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| *** masco is now known as masco|lunch | 09:11 | |
| *** lvdombrkr89 has joined #tripleo | 09:13 | |
| *** lvdombrkr has quit IRC | 09:15 | |
| *** etingof has quit IRC | 09:17 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates stable/pike: Fix puppet config volume for iscsid in containers https://review.openstack.org/531680 | 09:17 |
| *** pkovar has joined #tripleo | 09:18 | |
| *** suuuper has quit IRC | 09:19 | |
| *** aditya_r has quit IRC | 09:23 | |
| *** stendulker has joined #tripleo | 09:24 | |
| *** lucas-afk is now known as lucasagomes | 09:25 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix log file nova-conductor service https://review.openstack.org/532513 | 09:28 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: WIP ffu: Introduce Glance fast-forward upgrade tasks https://review.openstack.org/521544 | 09:30 |
| Tengu | hello there! Any ETA for Queens release? | 09:31 |
| *** jfrancoa has quit IRC | 09:32 | |
| *** derekh has joined #tripleo | 09:39 | |
| openstackgerrit | Daniel Alvarez proposed openstack/tripleo-heat-templates master: Add support for OVN Metadata Agent https://review.openstack.org/502943 | 09:40 |
| *** fzdarsky has quit IRC | 09:40 | |
| *** moshele has joined #tripleo | 09:40 | |
| *** dtantsur|afk is now known as dtantsur | 09:42 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: WIP ffu: Introduce Swift fast-forward upgrade tasks https://review.openstack.org/526642 | 09:45 |
| *** jfrancoa has joined #tripleo | 09:47 | |
| *** suuuper has joined #tripleo | 09:48 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: WIP ffu: Introduce Pacemaker fast-forward upgrade tasks https://review.openstack.org/523399 | 09:51 |
| *** mcornea has joined #tripleo | 09:51 | |
| *** etingof has joined #tripleo | 09:56 | |
| *** tosky has quit IRC | 09:56 | |
| *** nyechiel has quit IRC | 09:57 | |
| *** gfidente has joined #tripleo | 09:57 | |
| *** gfidente has quit IRC | 09:57 | |
| *** gfidente has joined #tripleo | 09:57 | |
| *** numans has quit IRC | 10:05 | |
| *** numans has joined #tripleo | 10:07 | |
| dtantsur | shardy: hey, mind reviewing the patch we were talking about? https://review.openstack.org/#/c/532495/ I'm stuck in rechecking for now, but it's already apparent that it works, and I've got a confirmation from the initial reporter that it fixes trhe problem | 10:07 |
| *** tosky has joined #tripleo | 10:08 | |
| shardy | dtantsur: Hey, sure added to my list | 10:08 |
| shardy | Tengu: Hi! See https://releases.openstack.org/queens/schedule.html | 10:09 |
| dtantsur | thnx | 10:09 |
| openstackgerrit | Luke Hinds proposed openstack/tripleo-heat-templates master: Implements AIDE Intrusion Detection System https://review.openstack.org/483028 | 10:10 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 10:10 |
| shardy | Tengu: note that some TripleO projects/repos follow the cycle trailing deadline, but we normally aim to release as close to the main OpenStack GA as possible | 10:10 |
| *** daidv has quit IRC | 10:10 | |
| *** masco|lunch is now known as masco | 10:12 | |
| *** yamahata has quit IRC | 10:14 | |
| *** salmankhan has joined #tripleo | 10:15 | |
| Tengu | shardy: ah, cool, thanks! | 10:16 |
| *** moshele has quit IRC | 10:18 | |
| *** nyechiel has joined #tripleo | 10:25 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: WIP ffu: Introduce Nova fast-forward upgrade tasks https://review.openstack.org/522921 | 10:29 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common master: Add missing support for the root_device property https://review.openstack.org/533145 | 10:31 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates stable/pike: [DO NOT MERGE] Testing upgrades O->P. https://review.openstack.org/529802 | 10:32 |
| openstackgerrit | Merged openstack/tripleo-ui stable/ocata: Imported Translations from Zanata https://review.openstack.org/533086 | 10:35 |
| openstackgerrit | Merged openstack/tripleo-ui stable/pike: Imported Translations from Zanata https://review.openstack.org/533088 | 10:35 |
| *** stendulker_ has joined #tripleo | 10:36 | |
| *** stendulker has quit IRC | 10:40 | |
| *** dmacpher has joined #tripleo | 10:40 | |
| *** gchamoul has quit IRC | 10:40 | |
| *** zoli has quit IRC | 10:40 | |
| *** stendulker_ has quit IRC | 10:44 | |
| *** aditya_r has joined #tripleo | 10:47 | |
| *** shardy has quit IRC | 10:50 | |
| *** shardy has joined #tripleo | 10:52 | |
| *** gchamoul has joined #tripleo | 10:52 | |
| *** zoli has joined #tripleo | 10:55 | |
| *** aditya_ra has joined #tripleo | 10:57 | |
| *** aditya_r has quit IRC | 10:59 | |
| *** akrivoka has joined #tripleo | 11:00 | |
| *** aditya_ra has quit IRC | 11:01 | |
| *** aditya_r has joined #tripleo | 11:01 | |
| *** jillr has quit IRC | 11:02 | |
| *** social has quit IRC | 11:03 | |
| *** social has joined #tripleo | 11:03 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 11:10 |
| *** aditya_r has quit IRC | 11:17 | |
| *** aditya_r has joined #tripleo | 11:17 | |
| *** aditya_r has quit IRC | 11:19 | |
| *** aditya_r has joined #tripleo | 11:19 | |
| *** suuuper has quit IRC | 11:19 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates master: Render NIC config templates with jinja2 https://review.openstack.org/523638 | 11:22 |
| *** pkovar has quit IRC | 11:25 | |
| *** shreshtha has quit IRC | 11:29 | |
| *** anilvenkata has quit IRC | 11:30 | |
| *** nyechiel has quit IRC | 11:31 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common master: Add missing support for the root_device property https://review.openstack.org/533145 | 11:36 |
| *** jkilpatr has quit IRC | 11:44 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-common master: Add UC backup Mistral workflow definition and actions https://review.openstack.org/517610 | 11:50 |
| Tengu | small question: what's the typical MTU an openstacin VM (instance) should have? I'm having some weird network issue, and I'm wondering if the root cause isn't some MTU thingy, either with the instance, or maybe at some other level, like compute interface or something like that. | 11:57 |
| openstackgerrit | Merged openstack/tripleo-common master: Removed incorrect user input comments https://review.openstack.org/532218 | 11:58 |
| *** pdeore has quit IRC | 11:58 | |
| *** dtantsur is now known as dtantsur|brb | 11:59 | |
| *** moshele has joined #tripleo | 12:00 | |
| *** apevec has joined #tripleo | 12:00 | |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-heat-templates master: Set Horizon default member role to 'Member' https://review.openstack.org/533165 | 12:01 |
| *** raildo has joined #tripleo | 12:02 | |
| openstackgerrit | Jaganathan Palanisamy proposed openstack/tripleo-common stable/pike: Removed incorrect user input comments https://review.openstack.org/533166 | 12:02 |
| *** morazi has quit IRC | 12:04 | |
| Tengu | found that - is it also valid for pike? ee460bc2289a | 12:04 |
| Tengu | rah | 12:04 |
| Tengu | https://docs.openstack.org/mitaka/networking-guide/config-mtu.html | 12:04 |
| *** ratailor has quit IRC | 12:08 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 12:10 |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| shardy | Tengu: that neutron interface is controlled via the NeutronGlobalPhysnetMtu parameter, see puppet/services/neutron-base.yaml in t-h-t | 12:10 |
| *** moshele has quit IRC | 12:11 | |
| Tengu | shardy: hmm ok. is that doc up-to-date and still meaningful? https://docs.openstack.org/neutron/latest/admin/config-mtu.html | 12:12 |
| Tengu | I think I might need to do some modification in my current deploy -.-' | 12:12 |
| Tengu | and I'm not 100% sure we actually hit an MTU issue. Well, that very thought makes me point the mtu | 12:13 |
| shardy | Tengu: AFAIK it's still relevant, the TripleO default is 1500, same as neutron | 12:13 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Add a hook to run a bash script at the end of the overcloud prep. https://review.openstack.org/533167 | 12:13 |
| Tengu | shardy: my instance MTU is 1450, that might match, indeed. | 12:14 |
| Tengu | and when we add up docker network in that, it might start to be a bit tricky. | 12:14 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Adds Workflow to List Available Roles https://review.openstack.org/509832 | 12:19 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Adds UpdateRoles Workflow https://review.openstack.org/515045 | 12:19 |
| *** aditya_r has quit IRC | 12:25 | |
| *** pblaho1 has quit IRC | 12:26 | |
| *** lucasagomes is now known as lucas-hungry | 12:26 | |
| *** jkilpatr has joined #tripleo | 12:26 | |
| *** jaganathan has quit IRC | 12:27 | |
| *** moshele has joined #tripleo | 12:33 | |
| *** pkovar has joined #tripleo | 12:39 | |
| *** janki has joined #tripleo | 12:41 | |
| *** pblaho1 has joined #tripleo | 12:43 | |
| *** jpich has quit IRC | 12:43 | |
| *** psachin has quit IRC | 12:45 | |
| *** jpena is now known as jpena|lunch | 12:49 | |
| *** hjensas has quit IRC | 12:51 | |
| *** abishop has joined #tripleo | 12:53 | |
| *** stendulker has joined #tripleo | 12:53 | |
| *** stendulker has quit IRC | 12:54 | |
| *** stendulker has joined #tripleo | 12:54 | |
| *** hjensas has joined #tripleo | 12:55 | |
| *** hjensas has quit IRC | 12:55 | |
| *** hjensas has joined #tripleo | 12:55 | |
| *** morazi has joined #tripleo | 12:59 | |
| *** pblaho1 has quit IRC | 12:59 | |
| *** pblaho has joined #tripleo | 12:59 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common master: Use ansible python api for ansible action https://review.openstack.org/511254 | 13:00 |
| *** jlabarre has joined #tripleo | 13:01 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-common master: Add UC backup Mistral workflow definition and actions https://review.openstack.org/517610 | 13:03 |
| *** Goneri has joined #tripleo | 13:05 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates stable/pike: Convert ipv4 nic-config templates https://review.openstack.org/533177 | 13:07 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates stable/ocata: Convert ipv4 nic-config templates https://review.openstack.org/533178 | 13:07 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 13:10 |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| *** hrybacki has quit IRC | 13:10 | |
| *** absubram has quit IRC | 13:11 | |
| *** hrybacki has joined #tripleo | 13:13 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Document ceph-ansible options used to deploy at scale https://review.openstack.org/531214 | 13:13 |
| weshay | mwhahaha, now scenario001 seems busted again | 13:15 |
| weshay | jebus | 13:15 |
| *** lucas-hungry is now known as lucasagomes | 13:16 | |
| *** nyechiel has joined #tripleo | 13:16 | |
| weshay | anyone see this issue? | 13:16 |
| weshay | "Error: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20180112-8-3eqd5h create failed: Error: 'ceph-cfg-dir' already exists", | 13:16 |
| weshay | "Error: /Stage[main]/Tripleo::Profile::Pacemaker::Cinder::Backup_bundle/Pacemaker::Resource::Bundle[openstack-cinder-backup]/Pcmk_bundle[openstack-cinder-backup]/ensure: change from absent to present failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20180112-8-3eqd5h create failed: Error: 'ceph-cfg-dir' already exists", | 13:16 |
| weshay | in scen001 | 13:17 |
| arxcruz|rover | weshay: seems to be related to https://bugs.launchpad.net/tripleo/+bug/1742795 | 13:17 |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 13:17 |
| arxcruz|rover | weshay: https://review.openstack.org/#/c/532986/ | 13:18 |
| weshay | arxcruz|rover++ | 13:18 |
| weshay | thanks | 13:18 |
| weshay | EmilienM++ | 13:19 |
| weshay | panda|ruck|afk, ^^^ | 13:19 |
| *** weshay changes topic to "CI status: RED (zuul restart lost jobs, scenario001/2 broken in master) | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest/" | 13:20 | |
| *** numans has quit IRC | 13:20 | |
| *** numans has joined #tripleo | 13:20 | |
| *** dprince has joined #tripleo | 13:21 | |
| *** moshele has quit IRC | 13:23 | |
| *** lblanchard has joined #tripleo | 13:25 | |
| *** dtantsur|brb is now known as dtantsur | 13:30 | |
| mwhahaha | Yea there is a fix for 001 | 13:31 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Default empty map for docker_config steps https://review.openstack.org/533185 | 13:32 |
| *** pgadiya has quit IRC | 13:36 | |
| *** trown|outtypewww is now known as trown | 13:36 | |
| *** ykarel is now known as ykarel|away | 13:41 | |
| openstackgerrit | Lukas Bezdicka proposed openstack/tripleo-heat-templates master: ffu: tripleo-packages repo management https://review.openstack.org/515429 | 13:41 |
| openstackgerrit | Lukas Bezdicka proposed openstack/tripleo-heat-templates master: ffu: Allow FASTFORWARDUPGRADE as a StackUpdateType https://review.openstack.org/522547 | 13:42 |
| openstackgerrit | Lukas Bezdicka proposed openstack/tripleo-heat-templates master: ffu: Add fast-forward-upgrade env https://review.openstack.org/522203 | 13:42 |
| *** cylopez has joined #tripleo | 13:43 | |
| *** rlandy has joined #tripleo | 13:44 | |
| *** jpena|lunch is now known as jpena | 13:49 | |
| *** nyechiel has quit IRC | 13:53 | |
| *** ykarel|away has quit IRC | 13:53 | |
| openstackgerrit | Sven Anderson proposed openstack/tripleo-heat-templates master: WIP: Add custom role for realtime compute https://review.openstack.org/531739 | 13:53 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Add a lookup plugin for nova servers https://review.openstack.org/514307 | 14:02 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Add lookup plugin for ironic nodes https://review.openstack.org/514308 | 14:02 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Add a health check for overcloud nodes https://review.openstack.org/510560 | 14:02 |
| *** rbowen has joined #tripleo | 14:09 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 14:10 |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| *** amoralej is now known as amoralej|lunch | 14:14 | |
| openstackgerrit | Merged openstack/tripleo-common master: Fail config-download when group:os-apply-config used https://review.openstack.org/531499 | 14:14 |
| openstackgerrit | Merged openstack/os-apply-config master: Updated from global requirements https://review.openstack.org/520470 | 14:14 |
| *** ecerquei has joined #tripleo | 14:16 | |
| *** florianf has quit IRC | 14:16 | |
| *** aputtur_ has joined #tripleo | 14:17 | |
| *** stendulker has quit IRC | 14:17 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-common master: Add UC Backup Mistral workflow definition and actions https://review.openstack.org/517610 | 14:22 |
| *** jmelvin has joined #tripleo | 14:22 | |
| *** liverpooler has joined #tripleo | 14:23 | |
| *** nyechiel has joined #tripleo | 14:25 | |
| *** liverpooler has quit IRC | 14:25 | |
| *** liverpooler has joined #tripleo | 14:25 | |
| *** dtrainor has quit IRC | 14:30 | |
| *** moshele has joined #tripleo | 14:30 | |
| *** numans has quit IRC | 14:32 | |
| *** moshele has quit IRC | 14:33 | |
| *** thrash|g0ne is now known as thrash | 14:35 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: kernel: allow to override modules & sysctl settings https://review.openstack.org/532735 | 14:35 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/520545 | 14:35 |
| *** toure|gone is now known as toure | 14:36 | |
| *** janki has quit IRC | 14:36 | |
| *** pkovar has quit IRC | 14:41 | |
| *** athomas has quit IRC | 14:42 | |
| *** athomas has joined #tripleo | 14:43 | |
| *** dtrainor has joined #tripleo | 14:43 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Default empty map for docker_config steps https://review.openstack.org/533185 | 14:49 |
| *** suuuper has joined #tripleo | 14:52 | |
| *** aditya_r has joined #tripleo | 14:55 | |
| *** ykarel|away has joined #tripleo | 14:55 | |
| EmilienM | hu | 14:59 |
| EmilienM | or hi even | 14:59 |
| *** amoralej|lunch is now known as amoralej | 14:59 | |
| EmilienM | weshay: yeah I fixed it | 15:00 |
| shardy | Hi EmilienM | 15:00 |
| weshay | thanks | 15:00 |
| EmilienM | weshay: https://review.openstack.org/#/c/532986/ | 15:00 |
| EmilienM | I think | 15:00 |
| EmilienM | weshay: to make sure it doesn't happen again, I improved the zuul layout | 15:00 |
| EmilienM | to run scenarios on pacemaker files | 15:00 |
| EmilienM | see https://review.openstack.org/#/c/532992/ | 15:01 |
| weshay | +1 | 15:01 |
| EmilienM | and https://review.openstack.org/#/c/532997/ | 15:01 |
| EmilienM | weshay: please review ^ | 15:01 |
| EmilienM | it's a CI thing | 15:01 |
| *** cylopez has quit IRC | 15:07 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742795 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742936 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1742795 in tripleo "Pacemaker fails to create openstack-cinder-backup bundle" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 15:10 |
| openstack | Launchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 15:10 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Add post_upgrade_tasks with post_upgrade_steps_playbook output https://review.openstack.org/511228 | 15:11 |
| EmilienM | has someone investigating on 1742936 ? ^ | 15:11 |
| EmilienM | bandini: hey, so did you find out something? | 15:11 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Remove dlrn_hash_tag_newest from upgrades job. https://review.openstack.org/529571 | 15:11 |
| bandini | EmilienM: yes I updated the review | 15:12 |
| EmilienM | bandini: I see enable_pacemaker: true | 15:12 |
| EmilienM | bandini: so why enable_pacemaker: false on other scenarios? I'm confused | 15:12 |
| bandini | EmilienM: ok so I did not look at the other scenarios (but if we claim that it is an HA job I presume we want pacemaker)? | 15:13 |
| EmilienM | yeah | 15:13 |
| EmilienM | bandini: anyway... thanks for your help | 15:13 |
| EmilienM | bandini: because it worked | 15:13 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs032: activate config-download https://review.openstack.org/528761 | 15:14 |
| bandini | EmilienM: so the reason it was not working is http://paste.openstack.org/show/643576/ (basically we use keepalived and not pacemaker and keepalived is not running on the controller - for reasons i did not investigate) | 15:14 |
| *** marrusl has quit IRC | 15:14 | |
| EmilienM | bandini: because of https://review.openstack.org/#/c/532623/4/ci/environments/multinode-3nodes-registry.yaml ? | 15:16 |
| EmilienM | OS::TripleO::Services::Keepalived: OS::Heat::None | 15:16 |
| bandini | EmilienM: ah yes that would explain why no keepalived | 15:17 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fixes generating public certificates https://review.openstack.org/532565 | 15:17 |
| openstackgerrit | Merged openstack/tripleo-common master: Validate parameters before updating https://review.openstack.org/511249 | 15:17 |
| *** panda|ruck|afk is now known as panda|ruck | 15:17 | |
| EmilienM | bandini: what confuses me is that scenarios don't have pacemaker enabled | 15:18 |
| *** yamahata has joined #tripleo | 15:18 | |
| EmilienM | http://logs.openstack.org/45/533145/2/check/tripleo-ci-centos-7-containers-multinode/9bde2be/logs/subnode-2/var/log/extra/dump_variables_hostvars.json.txt.gz | 15:18 |
| EmilienM | "enable_pacemaker": false, | 15:18 |
| EmilienM | but keepalived disabled: https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/multinode-containers.yaml#L10 | 15:18 |
| EmilienM | so wtf? | 15:18 |
| EmilienM | bandini: you see what I mean? | 15:18 |
| bandini | yeah that's concerning | 15:19 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Start step at 0 for update_ + upgrade_steps_playbook https://review.openstack.org/532147 | 15:19 |
| EmilienM | bandini: I'll try to set enable_pacemaker: true for all scenarios and see how that works | 15:20 |
| EmilienM | bandini: anyway, thanks again for looking at it, I scratched my head a few hours on that :D | 15:20 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Validate roles data and network data https://review.openstack.org/508567 | 15:20 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add a Get Networks workflow https://review.openstack.org/509419 | 15:20 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add UpdateNetworks workflow https://review.openstack.org/513463 | 15:20 |
| EmilienM | bandini: tbh the fact we fixed the logs recently really helped | 15:21 |
| bandini | EmilienM: so we should set it to true to all those that don't have the 'x' on the nonHA column, correct? | 15:21 |
| bandini | EmilienM: it took me some time as well to understand wtf was going on :) | 15:21 |
| EmilienM | corrcetly | 15:21 |
| *** cylopez has joined #tripleo | 15:22 | |
| bandini | ack,I assumed that was the case already so I did not compare | 15:22 |
| EmilienM | weshay, arxcruz|rover : the Tempest issue in promotion pipeline is tempest not able to SSH the VM: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/b09f8f0/undercloud/home/jenkins/tempest_output.log.txt.gz#_2018-01-11_12_50_11 | 15:22 |
| EmilienM | I'm looking at it now. | 15:22 |
| arxcruz|rover | EmilienM: haleyb is also looking at it, according dalvarez this is a bug in neutron | 15:23 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Adds Workflow to List Available Roles https://review.openstack.org/509832 | 15:26 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Adds UpdateRoles Workflow https://review.openstack.org/515045 | 15:26 |
| EmilienM | no red in the logs (neutron, nova, cinder) | 15:26 |
| EmilienM | arxcruz|rover, dalvarez : why neutron? show me logs | 15:27 |
| arxcruz|rover | EmilienM: dunno, dalvarez is the guy :) | 15:27 |
| *** jd_ has quit IRC | 15:27 | |
| EmilienM | I see the VM created & interfaces attached | 15:28 |
| EmilienM | at that point it might be some metadata issue | 15:28 |
| EmilienM | ihrachys: you around? | 15:29 |
| ihrachys | meeting for next hour | 15:29 |
| EmilienM | k | 15:29 |
| arxcruz|rover | EmilienM: yes, as I mention in the bug, this seems to be similar to the previous one we had, the vm tries to ssh, but in that time, the sshd service isn't started yet, as it starts after metadata service | 15:29 |
| *** pkovar has joined #tripleo | 15:29 | |
| arxcruz|rover | then it times out on metadata, sshd starts, but the keys weren't injected | 15:30 |
| arxcruz|rover | so you get authentication error | 15:30 |
| EmilienM | which sshd service? | 15:30 |
| *** dpawar has quit IRC | 15:30 | |
| EmilienM | the cirros image has sshd service started when it boots | 15:30 |
| weshay | panda|ruck, ^ | 15:30 |
| EmilienM | and it's a timeout which means the service *tried* a bunch of time | 15:30 |
| arxcruz|rover | EmilienM: the sshd service in cirros starts after tries the metadata | 15:31 |
| EmilienM | it actually tried 20 seconds | 15:31 |
| *** dansmith is now known as superdan | 15:31 | |
| *** aufi_ has quit IRC | 15:31 | |
| EmilienM | arxcruz|rover: where do you see that? | 15:31 |
| EmilienM | and so in that case, why would it be neutron? maybe the VM is just insanely slow | 15:32 |
| panda|ruck | is the metadata taken from a url on a link-local address ? | 15:32 |
| dalvarez | EmilienM, arxcruz|rover yeah not sure, last time i saw some errors in l3 agent and i figured out that metadata rules were missing in iptables in qrouter namespace | 15:33 |
| arxcruz|rover | 2018-01-11 12:48:50,714 12674 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.108 ([Errno None] Unable to connect to port 22 on 192.168.24.108). Number attempts: 19. Retry after 20 seconds. | 15:33 |
| arxcruz|rover | 2018-01-11 12:49:11,283 12674 INFO [paramiko.transport] Connected (version 2.0, client dropbear_2012.55) | 15:33 |
| arxcruz|rover | 2018-01-11 12:49:11,365 12674 INFO [paramiko.transport] Authentication (publickey) failed. | 15:33 |
| dalvarez | EmilienM, arxcruz|rover but this time i cant see those errors, though i dont see metadata requests from the VMs neither in the neutron metadata agent nor in nova metadata api | 15:33 |
| dalvarez | EmilienM, arxcruz|rover last time we worked it around with ovsdbapp but the root cause was still there and haleyb took care of that even though his fixes are not yet merged | 15:34 |
| mwhahaha | EmilienM: Should we revert the addition of scenario002 for THT (as it's in the gate non-voting) | 15:34 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Revert "Re-enable voting on scenario002-container" https://review.openstack.org/533213 | 15:35 |
| *** lvdombrkr89 has quit IRC | 15:35 | |
| mwhahaha | weshay, EmilienM -^ | 15:35 |
| EmilienM | mwhahaha: meh | 15:36 |
| EmilienM | but ok | 15:36 |
| mwhahaha | unless scenario002 is going to get fixed today | 15:36 |
| *** haleyb has joined #tripleo | 15:36 | |
| *** jtomasek has quit IRC | 15:36 | |
| *** dpawar has joined #tripleo | 15:37 | |
| EmilienM | dalvarez: tbh, I can't help much on that one | 15:37 |
| dalvarez | yeah | 15:37 |
| dalvarez | EmilienM, haleyb arxcruz|rover last time it was super useful for me that someone from tipleo would put some node on hold so that i could connect and dig | 15:38 |
| *** suuuper has quit IRC | 15:38 | |
| *** fragatina has joined #tripleo | 15:38 | |
| EmilienM | dalvarez, arxcruz|rover: is it happening only on this featureset? | 15:38 |
| *** suuuper has joined #tripleo | 15:38 | |
| holser__ | Good morning EmilienM - Could you put some thoughts into https://review.openstack.org/#/c/530793/2 ? | 15:38 |
| arxcruz|rover | EmilienM: in two multinode jobs, let me check | 15:38 |
| arxcruz|rover | dalvarez: well, right now rdocloud isn't stable to get a node for you... | 15:39 |
| arxcruz|rover | but we can work on that | 15:39 |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add Openstack Undercloud backup https://review.openstack.org/466213 | 15:39 |
| EmilienM | weshay: any thoughts on https://review.openstack.org/#/c/530793/2/roles/provision/support_check/tasks/main.yml ? | 15:39 |
| hamzy | I'm trying to do an inspection of a node with tripleo and am seeing the following "ConnectionError: HTTPConnectionPool(host='192.168.0.253', port=5050): Max retries exceeded with url" The full log is at: https://hamzy.fedorapeople.org/TripleO-multi-arch/deploy-c365f02u23.fail | 15:41 |
| *** marrusl has joined #tripleo | 15:41 | |
| arxcruz|rover | EmilienM: yes, only on this featureset | 15:41 |
| *** yprokule has quit IRC | 15:41 | |
| arxcruz|rover | sorry, the other is the telemetry error | 15:41 |
| *** fragatina has quit IRC | 15:42 | |
| EmilienM | ok so it's only when ceph is enabled | 15:42 |
| *** dpawar has quit IRC | 15:42 | |
| *** mcornea has quit IRC | 15:42 | |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add Openstack Undercloud backup https://review.openstack.org/466213 | 15:42 |
| EmilienM | maybe it's a problem with performance | 15:42 |
| hamzy | I am seeing the node get an address "dhclient[5585]: bound to 9.114.118.249 -- renewal in 254 seconds" | 15:43 |
| *** Goneri has quit IRC | 15:44 | |
| hamzy | ah... I think I know what that got it | 15:44 |
| weshay | EmilienM, I'll comment in a few | 15:45 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Create Swift directory d1 if needed https://review.openstack.org/517373 | 15:46 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Create Swift directory d1 if required https://review.openstack.org/517374 | 15:46 |
| openstackgerrit | Merged openstack/tripleo-common stable/pike: DPDK derive params inputs default value update https://review.openstack.org/529036 | 15:46 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/pike: Bind-mount iscsid IQN by its real host path https://review.openstack.org/531679 | 15:46 |
| openstackgerrit | Merged openstack/puppet-tripleo master: zuul: run scenarios when patching pacemaker profiles https://review.openstack.org/532992 | 15:46 |
| openstackgerrit | Merged openstack/puppet-tripleo master: cinder/pacemaker: resolve puppet resource duplications https://review.openstack.org/532986 | 15:46 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/newton: kernel: allow to override modules & sysctl settings https://review.openstack.org/532736 | 15:46 |
| EmilienM | woot | 15:50 |
| EmilienM | one alert down | 15:50 |
| mwhahaha | mergey mergey | 15:50 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Adds fast_forward_upgrade_tasks for Heat services https://review.openstack.org/530956 | 15:56 |
| *** dparkes has quit IRC | 16:00 | |
| *** suuuper has quit IRC | 16:03 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates stable/pike: Add noop's for docker pacemaker for rabbitmq https://review.openstack.org/533221 | 16:04 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/pike: zuul: run scenarios when touching pacemaker services https://review.openstack.org/533057 | 16:07 |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Add condition to create container images download script. https://review.openstack.org/532772 | 16:07 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/pike: cinder/pacemaker: resolve puppet resource duplications https://review.openstack.org/533058 | 16:07 |
| openstackgerrit | Merged openstack/tripleo-common master: Add ansible playbook for configuring octavia https://review.openstack.org/487450 | 16:07 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742936 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 16:10 |
| *** itlinux has joined #tripleo | 16:12 | |
| *** cylopez has quit IRC | 16:12 | |
| *** cylopez has joined #tripleo | 16:14 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-common master: Add UC Backup Mistral workflow definition and actions https://review.openstack.org/517610 | 16:14 |
| mandre | pradk: what replaces python-ceilometerclient? | 16:17 |
| pradk | mandre, nothing | 16:17 |
| pradk | well | 16:17 |
| pradk | ceilometer api is dead | 16:17 |
| pradk | so we dont really need ceiloclient | 16:17 |
| pradk | instead use aodh, gnocchi, panko for various services | 16:18 |
| pradk | and their respective clients | 16:18 |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-common master: Update Ansible playbook for Swift to add all missing devices to rings https://review.openstack.org/529708 | 16:19 |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-common master: Make Swift playbook usable without tripleo-ansible-inventory https://review.openstack.org/533233 | 16:19 |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates master: Add Ansible-based Swift ring deployment environment https://review.openstack.org/533234 | 16:19 |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates master: Enable Ansible-based Swift ring deployment by default https://review.openstack.org/533235 | 16:19 |
| *** itlinux has quit IRC | 16:20 | |
| *** ebarrera has quit IRC | 16:21 | |
| *** ykarel|away has quit IRC | 16:21 | |
| mandre | pradk: ack, thanks for the explanation | 16:22 |
| *** dmacpher has quit IRC | 16:24 | |
| *** jd_ has joined #tripleo | 16:27 | |
| openstackgerrit | Sven Anderson proposed openstack/tripleo-image-elements master: Add realtime-kernel element https://review.openstack.org/531442 | 16:35 |
| *** jistr has quit IRC | 16:37 | |
| *** dtantsur has quit IRC | 16:37 | |
| *** jwb has quit IRC | 16:37 | |
| *** leseb has quit IRC | 16:37 | |
| *** StevenK has quit IRC | 16:37 | |
| *** ansiwen has quit IRC | 16:37 | |
| *** leseb has joined #tripleo | 16:37 | |
| *** StevenK has joined #tripleo | 16:37 | |
| *** jwb has joined #tripleo | 16:38 | |
| *** links has quit IRC | 16:38 | |
| *** itlinux has joined #tripleo | 16:39 | |
| *** jistr has joined #tripleo | 16:40 | |
| *** ansiwen has joined #tripleo | 16:40 | |
| ansiwen | arxcruz|rover: can you tell me, where I find the tempestconf output here: https://logs.rdoproject.org/14/519714/10/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset021-master/Z6f65d955b2ba4352a08df5767141bbd5/ | 16:42 |
| *** links has joined #tripleo | 16:42 | |
| ansiwen | arxcruz|rover: ^^^ | 16:43 |
| arxcruz|rover | ansiwen: https://logs.rdoproject.org/14/519714/10/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset021-master/Z6f65d955b2ba4352a08df5767141bbd5/undercloud/home/jenkins/tempest_output.log.txt.gz#_2018-01-12_16_02_27 | 16:44 |
| ansiwen | arxcruz, thanks a lot, obviously I'm blind | 16:45 |
| arxcruz|rover | ;) | 16:45 |
| -openstackstatus- NOTICE: Zuul has been restarted and lost queue information; changes in progress will need to be rechecked. | 16:46 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Use docker_config_scripts for puppet apply https://review.openstack.org/533062 | 16:47 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Use docker_config_scripts for puppet apply https://review.openstack.org/533062 | 16:47 |
| *** mdnadeem has quit IRC | 16:48 | |
| *** ecerquei has quit IRC | 16:50 | |
| EmilienM | marios: I didn't know we have to wait for upper stable backports to land, see https://review.openstack.org/#/c/532981/ | 16:50 |
| mwhahaha | god damn it they lost the queue again | 16:51 |
| EmilienM | mandre: usually backports land asynchronously | 16:51 |
| EmilienM | mwhahaha: my fault again? | 16:51 |
| *** ecerquei has joined #tripleo | 16:51 | |
| *** ecerquei has quit IRC | 16:51 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Correct panko auth region https://review.openstack.org/532980 | 16:52 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Add DockerPuppetDebug to environments/config-debug.yaml https://review.openstack.org/532728 | 16:52 |
| *** fragatina has joined #tripleo | 16:54 | |
| EmilienM | cschwede: https://review.openstack.org/#/c/517374/ do we need to backport in pike? | 16:55 |
| EmilienM | seems like a queen thing, right? | 16:55 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart master: [DNM] Temporary hack to go over the lack of promotion. https://review.openstack.org/533242 | 16:57 |
| *** cylopez has quit IRC | 16:58 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates stable/pike: Fix path for iscsi config file https://review.openstack.org/532523 | 16:59 |
| *** absubram has joined #tripleo | 17:00 | |
| *** pcaruana has quit IRC | 17:03 | |
| *** anilvenkata has joined #tripleo | 17:04 | |
| *** fragatina has quit IRC | 17:07 | |
| *** fragatina has joined #tripleo | 17:07 | |
| *** dciabrin_ has joined #tripleo | 17:09 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742936 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 17:10 |
| *** Goneri has joined #tripleo | 17:10 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart master: New featureset for fast forward upgrade testing. https://review.openstack.org/525686 | 17:10 |
| *** dciabrin has quit IRC | 17:11 | |
| *** tesseract has quit IRC | 17:12 | |
| shardy | Hey all, looks like the gate may be broken from $status, but when ready https://review.openstack.org/#/c/532768/ and https://review.openstack.org/#/c/532811/ are ready for reviews, thanks! | 17:13 |
| *** cylopez has joined #tripleo | 17:19 | |
| *** links has quit IRC | 17:24 | |
| *** shardy has quit IRC | 17:28 | |
| marios | EmilienM: ack i'll check but next week thanks | 17:30 |
| *** marios has quit IRC | 17:30 | |
| EmilienM | bogdando: you shouldn't depend-on on patches on the same project | 17:31 |
| EmilienM | e.g. https://review.openstack.org/#/c/531680/ | 17:31 |
| EmilienM | it can produce issues when zuul clones repo | 17:31 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Fix puppet config volume for iscsid in containers https://review.openstack.org/531680 | 17:31 |
| *** gfidente is now known as gfidente|afk | 17:32 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Fix puppet config volume for iscsid in containers https://review.openstack.org/531680 | 17:32 |
| *** bogdando has quit IRC | 17:33 | |
| *** holser__ has quit IRC | 17:34 | |
| *** absubram has quit IRC | 17:34 | |
| *** cshastri has quit IRC | 17:36 | |
| *** masco has quit IRC | 17:42 | |
| bnemec | $ ping github.com | 17:53 |
| bnemec | ping: github.com: Name or service not known | 17:53 |
| bnemec | Come on, Red Hat DNS. | 17:53 |
| bnemec | That's the second time this week. | 17:54 |
| EmilienM | bnemec: use RDU | 17:54 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Add a hook to run a bash script at the end of the overcloud prep. https://review.openstack.org/533167 | 17:54 |
| bnemec | EmilienM: Really? We get different DNS servers for different VPN endpoints? | 17:55 |
| *** trown is now known as trown|lunch | 17:56 | |
| *** gkadam has quit IRC | 17:58 | |
| *** derekh has quit IRC | 17:58 | |
| EmilienM | bnemec: yes | 17:59 |
| bnemec | Weird | 17:59 |
| *** anilvenkata has quit IRC | 17:59 | |
| bnemec | So how are container images supposed to be built now? The docs are completely broken and seem to refer to deprecated files. | 18:01 |
| *** lucasagomes is now known as lucas-afk | 18:07 | |
| *** dougbtv_ has quit IRC | 18:10 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742936 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 18:10 |
| EmilienM | arxcruz|rover, dalvarez^ before the weekend, any last update on this one? is anyone looking at it presently? | 18:11 |
| arxcruz|rover | EmilienM: nope | 18:12 |
| arxcruz|rover | have a good weekend :) | 18:12 |
| EmilienM | well, this is a bit concerning | 18:12 |
| arxcruz|rover | EmilienM: sorry, I can try to take a look, but I'm not good in this metadata issue :/ | 18:14 |
| EmilienM | arxcruz|rover: my question is more "is someone assigned to this task now" | 18:15 |
| arxcruz|rover | EmilienM: haleyb is looking at it | 18:15 |
| EmilienM | ok | 18:15 |
| EmilienM | ihrachys: when you're free also ^ | 18:15 |
| *** pkovar has quit IRC | 18:16 | |
| *** aditya_ra has joined #tripleo | 18:16 | |
| haleyb | arxcruz|rover: i've looked at logs but it's hard to say its metadata. is the job constantly failing or just intermittent? | 18:17 |
| arxcruz|rover | haleyb: constantly | 18:17 |
| cschwede | EmilienM: right, no backport for pike needed, will only affects queens | 18:18 |
| EmilienM | cschwede: ack, thx | 18:18 |
| *** aditya_r has quit IRC | 18:19 | |
| haleyb | arxcruz|rover: is it possible to lock a test system when it fails so i can login? the next thing i'm trying to find is the console for the instance | 18:20 |
| *** athomas has quit IRC | 18:20 | |
| ihrachys | haleyb, Authentication (publickey) failed so meaning keypair not configured | 18:20 |
| ihrachys | first thing I notice is the test doesn't dump console | 18:21 |
| ihrachys | we should make it | 18:21 |
| EmilienM | yeah, is it a parameter in tempest? I can't recall | 18:21 |
| haleyb | ihrachys: ok, so the IP is configured at least | 18:21 |
| arxcruz|rover | haleyb: I'll try to recreate it for you | 18:22 |
| haleyb | arxcruz|rover: thanks | 18:23 |
| ihrachys | EmilienM, no it needs test changes | 18:23 |
| ihrachys | which actually sucks and should be automatic | 18:23 |
| ihrachys | like 'dump consoles for all instances tracked before raising' | 18:23 |
| *** panda|ruck has quit IRC | 18:23 | |
| ihrachys | just catching it somwhere on base level and dumping would be nice | 18:24 |
| *** panda has joined #tripleo | 18:25 | |
| ihrachys | I wonder if there is a way to determine in cleanup function whether we failed | 18:26 |
| ihrachys | then we could add self.addCleanup in create_server that would dump console, and put it before delete_server cleanup function | 18:26 |
| ihrachys | ok I think this is the solution: https://stackoverflow.com/questions/12290336/how-to-execute-code-only-on-test-failures-with-python-unittest2 | 18:29 |
| *** GheRivero has joined #tripleo | 18:29 | |
| *** ChanServ sets mode: +v GheRivero | 18:29 | |
| *** cylopez has quit IRC | 18:33 | |
| *** slagle has quit IRC | 18:34 | |
| *** salmankhan has quit IRC | 18:34 | |
| *** slagle has joined #tripleo | 18:34 | |
| *** jpena is now known as jpena|off | 18:35 | |
| *** bandini has quit IRC | 18:36 | |
| *** bandini has joined #tripleo | 18:39 | |
| *** apevec has quit IRC | 18:40 | |
| ihrachys | haleyb, besides missing console log, any other ideas? | 18:43 |
| *** jfrancoa has quit IRC | 18:44 | |
| haleyb | ihrachys: debug = True everywhere | 18:44 |
| ihrachys | it's not? | 18:45 |
| haleyb | i didnn't think so based on log output let me look in /etc | 18:45 |
| ihrachys | haleyb, ok. where are the logs btw? can't find them. | 18:46 |
| haleyb | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/b09f8f0/ | 18:46 |
| haleyb | debug is set in neutron.conf but not l3-agent, unless it's started with that | 18:47 |
| haleyb | ihrachys: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/b09f8f0/subnode-2/var/log/extra/docker/containers/ | 18:47 |
| ihrachys | haleyb, I mean where's neutron | 18:47 |
| ihrachys | oh ok, thanks | 18:47 |
| haleyb | yeah, i guess l3-agent does have debug, there just wasn't much to see | 18:48 |
| *** jkilpatr has quit IRC | 18:49 | |
| *** dtantsur has joined #tripleo | 18:50 | |
| *** dougbtv has joined #tripleo | 18:55 | |
| ihrachys | haleyb, I guess we won't see metadata requests on proxy side either since we don't retain those logs. | 18:56 |
| ihrachys | and metadata agent is just plain empty after startup | 18:57 |
| haleyb | ihrachys: yeah, there's no data to track the metadata requests | 18:57 |
| haleyb | the console would at least tell us it tried | 18:59 |
| ihrachys | right. but isn't it a flow itself that we have nothing? | 18:59 |
| ihrachys | *flaw | 18:59 |
| ihrachys | I mean... it could be as well some random reactos VM logging errors to NSA servers, and we won't know whether it attempted to contact us | 19:00 |
| ihrachys | does haproxy log to syslog? | 19:00 |
| *** jkilpatr has joined #tripleo | 19:02 | |
| haleyb | the config file shows /dev/log i don't know where that ends up | 19:02 |
| haleyb | man page says syslog | 19:03 |
| ihrachys | there are some logs for haproxy requests in journal but those are for services not the haproxy we start for metadata | 19:03 |
| ihrachys | here is journal: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/b09f8f0/subnode-2/var/log/journal.txt.gz | 19:03 |
| ihrachys | so yeah, we don't have anything there either | 19:06 |
| ihrachys | maybe we should make haproxy log there | 19:06 |
| ihrachys | then we would have requests seen from instance (or not) | 19:06 |
| ihrachys | hm we actually configure it for option httplog | 19:09 |
| ihrachys | which should make it log all requests | 19:09 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742936 | 19:10 |
| openstack | Launchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| haleyb | ihrachys: where would it log to? | 19:12 |
| *** dhill__ has joined #tripleo | 19:13 | |
| ihrachys | /dev/log whatever it is | 19:14 |
| ihrachys | also, quoting from the internet | 19:14 |
| ihrachys | "the main problem is that the chrooted haproxy won't be able to access /dev/log and in order to circumvent the issue you can either: | 19:14 |
| ihrachys | Enable syslog to listen on the UDP socket (usually on port 514) as described in the other messages." | 19:14 |
| ihrachys | could it be that there is no /dev/log for haproxy to log into when in container? | 19:15 |
| ihrachys | see here too: https://github.com/dockerfile/haproxy/issues/3 | 19:16 |
| *** amoralej is now known as amoralej|off | 19:16 | |
| ihrachys | so it's either we configure it to log via network to syslog daemon (I assume systemd provides one and enables it) | 19:17 |
| ihrachys | or we need to expose /dev/log into container | 19:17 |
| ihrachys | (I assume we don't) | 19:17 |
| ihrachys | considering that the former would require non-obvious changes to neutron itself, I would try the later | 19:17 |
| ihrachys | *latter | 19:17 |
| openstackgerrit | OpenStack Release Bot proposed openstack/tripleo-upgrade stable/pike: Update .gitreview for stable/pike https://review.openstack.org/533275 | 19:18 |
| *** moshele has joined #tripleo | 19:18 | |
| openstackgerrit | OpenStack Release Bot proposed openstack/tripleo-upgrade stable/pike: Update UPPER_CONSTRAINTS_FILE for stable/pike https://review.openstack.org/533276 | 19:18 |
| openstackgerrit | OpenStack Release Bot proposed openstack/tripleo-upgrade master: Update reno for stable/pike https://review.openstack.org/533277 | 19:18 |
| ihrachys | haleyb, I guess it would go somewhere https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/neutron-l3.yaml | 19:19 |
| ihrachys | and of course same for dhcp agent since it can also start metadata | 19:19 |
| *** trown|lunch is now known as trown | 19:22 | |
| haleyb | ihrachys: i see a /dev/log entry in containers-common.yaml | 19:23 |
| ihrachys | haleyb, link | 19:24 |
| haleyb | https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/containers-common.yaml | 19:24 |
| haleyb | i would assume that applies everywhere? | 19:24 |
| ihrachys | yeah... | 19:25 |
| ihrachys | it's included in l3 agent definition | 19:25 |
| ihrachys | there is also a separate 'syslog' log: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/b09f8f0/subnode-2/syslog.txt.gz | 19:26 |
| ihrachys | not sure why it's separate | 19:26 |
| *** moshele has quit IRC | 19:26 | |
| ihrachys | bit it doesn't have anything of interest either | 19:26 |
| ihrachys | ok I used my pool of time for this issue for today. but I think the idea would be first to make those missing logging bits available - console and haproxy | 19:28 |
| ihrachys | also since proxy redirects some requests to metadata-agent, maybe we can have a look at why agent doesn't log passed through requests either | 19:28 |
| ihrachys | please start capturing details we have so far in the bug | 19:29 |
| ihrachys | and btw to aleviate the impact can we disable the test case for now somehow? | 19:29 |
| ihrachys | haleyb, are you going to track this one? | 19:30 |
| haleyb | ihrachys: i will update the bug and see about at least trying to disable for now, i don't have a lot of time left today either | 19:31 |
| *** jillr has joined #tripleo | 19:32 | |
| ihrachys | yeah. maybe we should drop a mail to european folks to look at it start of their Mon | 19:32 |
| *** moshele has joined #tripleo | 19:37 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates stable/pike: Fix log file nova-conductor service https://review.openstack.org/533282 | 19:37 |
| *** ecerquei has joined #tripleo | 19:38 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: zuul: run scenarios when touching pacemaker services https://review.openstack.org/533054 | 19:45 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: zuul: run scenarios when touching pacemaker services https://review.openstack.org/532997 | 19:45 |
| *** aditya_ra has quit IRC | 19:46 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart master: New featureset for fast forward upgrade testing. https://review.openstack.org/525686 | 19:48 |
| *** dsneddon is now known as dsneddon_afk | 19:53 | |
| *** eck` is now known as eck`gone | 19:55 | |
| *** eck`gone is now known as eck` | 19:56 | |
| openstackgerrit | Merged openstack/python-tripleoclient master: Revert "Revert "Convert undercloud deploy to use TripleoInventory class"" https://review.openstack.org/532768 | 20:06 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742936 | 20:10 |
| openstack | Launchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| *** yamahata has quit IRC | 20:10 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Add DockerPuppetDebug to environments/config-debug.yaml https://review.openstack.org/532728 | 20:24 |
| *** fragatina has quit IRC | 20:38 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: New featureset for fast forward upgrade testing. https://review.openstack.org/525686 | 20:42 |
| weshay | chem, posted doc for fs36 | 20:43 |
| *** yamahata has joined #tripleo | 20:45 | |
| *** openstackstatus has quit IRC | 20:46 | |
| *** openstack has joined #tripleo | 20:47 | |
| *** ChanServ sets mode: +o openstack | 20:47 | |
| slagle | trown: around? can you review https://review.openstack.org/#/c/531221/ | 20:49 |
| *** openstackstatus has joined #tripleo | 20:49 | |
| *** ChanServ sets mode: +v openstackstatus | 20:49 | |
| slagle | it should be good to go | 20:49 |
| slagle | well, i guess i didnt try it with newton | 20:51 |
| slagle | dang, i always forget about that crusty fella | 20:51 |
| weshay | slagle, I think they took their last import from newton yesterday | 20:52 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Validate roles data and network data https://review.openstack.org/508567 | 20:53 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add a Get Networks workflow https://review.openstack.org/509419 | 20:53 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add UpdateNetworks workflow https://review.openstack.org/513463 | 20:53 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates stable/newton: DO NOT MERGE - test nic-templates for stable/newton https://review.openstack.org/533297 | 20:53 |
| trown | slagle: ya it is not there in newton https://github.com/openstack/tripleo-heat-templates/tree/stable/newton/ci/environments | 20:54 |
| slagle | so it's going to break newton ovb jobs | 20:54 |
| *** dougbtv has quit IRC | 20:54 | |
| trown | slagle: ya :( ... so one way is to put an {% if release != 'newton' %} like we do for the telemetry args | 20:56 |
| slagle | should i backport it all to newton as well, or we could "if newton" quickstart | 20:56 |
| slagle | that might be easiest | 20:56 |
| trown | ya I think that is better than backporting | 20:56 |
| *** rbowen has quit IRC | 20:58 | |
| *** paramite has quit IRC | 21:00 | |
| *** itlinux has quit IRC | 21:02 | |
| openstackgerrit | Carol Bouchard proposed openstack/tripleo-heat-templates master: Latest Nexus Configuration Variables Updates https://review.openstack.org/527508 | 21:04 |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart master: Use network templates from tripleo-heat-templates https://review.openstack.org/531221 | 21:05 |
| *** paramite has joined #tripleo | 21:06 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742936 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 21:10 |
| *** moshele has quit IRC | 21:10 | |
| *** gfidente|afk has quit IRC | 21:17 | |
| weshay | mwhahaha, the neutron team is suggesting we disable volumebootpattern | 21:17 |
| weshay | or considering it | 21:18 |
| mwhahaha | Because.... | 21:18 |
| *** dhill__ has quit IRC | 21:19 | |
| *** paramite has quit IRC | 21:19 | |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Adds Workflow to List Available Roles https://review.openstack.org/509832 | 21:20 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Adds UpdateRoles Workflow https://review.openstack.org/515045 | 21:20 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Select Roles Workflow https://review.openstack.org/528282 | 21:20 |
| *** liverpooler has quit IRC | 21:21 | |
| *** chlong_ has joined #tripleo | 21:22 | |
| *** chlong has quit IRC | 21:25 | |
| haleyb | weshay: i just threw that out there, don't expect it to merge | 21:25 |
| haleyb | there's just not enough data right now to know why it fails | 21:26 |
| weshay | haleyb, which reliew? | 21:29 |
| weshay | review | 21:29 |
| weshay | rather | 21:29 |
| *** salmankhan has joined #tripleo | 21:29 | |
| *** rlandy has quit IRC | 21:29 | |
| haleyb | https://review.openstack.org/533298 | 21:29 |
| haleyb | but it's the bug you commented in it's referring to, if i can reproduce it downstream that would be good | 21:30 |
| *** itlinux has joined #tripleo | 21:32 | |
| weshay | haleyb, so for tripleo you have the option of updating https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/vars/tempest_skip_master.yml | 21:39 |
| weshay | as well | 21:39 |
| *** eggmaster has joined #tripleo | 21:39 | |
| weshay | haleyb, I sent you the recreate script in the bug | 21:40 |
| haleyb | weshay: yes, that would be a better way to disable just for tripleo. the test seems to run fine in the neutron jobs so i don't know what is different | 21:42 |
| weshay | haleyb, well.. we'd have to get mwhahaha and EmilienM to approve. It's one of just a few tests we have the time to run in each job. So.. I'm not optimistic | 21:43 |
| weshay | haleyb, I am pretty confident we can get a recreate env up though | 21:43 |
| weshay | haleyb, do you have an rdo-cloud account? | 21:43 |
| haleyb | weshay: yes, that is best | 21:44 |
| *** ecerquei_ has joined #tripleo | 21:44 | |
| *** ecerquei has quit IRC | 21:44 | |
| haleyb | yes, but i need to leave soon | 21:44 |
| EmilienM | ignore a test like this in our CI would be a huge regression | 21:45 |
| weshay | EmilienM, ya.. agree | 21:45 |
| EmilienM | it's just spawning a VM (from volume) and trying a ping | 21:45 |
| EmilienM | strong -2 | 21:45 |
| weshay | STRONG | 21:45 |
| *** holser__ has joined #tripleo | 21:45 | |
| weshay | LARGE | 21:45 |
| EmilienM | because if we open it, then we'll probably merge patches that break us | 21:45 |
| weshay | agree | 21:45 |
| haleyb | i didn't know if it was blocking something more important | 21:45 |
| EmilienM | it works in devstack, good, but not for us | 21:46 |
| EmilienM | so no promotion until it works | 21:46 |
| weshay | haleyb, it's blocking OSP13 builds | 21:46 |
| EmilienM | yeah but no pression :D | 21:46 |
| weshay | haleyb, the fate of the free world is in your hands | 21:46 |
| EmilienM | lol | 21:47 |
| EmilienM | indeed | 21:47 |
| haleyb | hah, too late for the free world | 21:47 |
| weshay | I think we scared Brian off | 21:47 |
| weshay | oh .. we have a hero | 21:47 |
| EmilienM | :) | 21:48 |
| weshay | I'll get an env going too jic | 21:50 |
| EmilienM | weshay: thx | 21:51 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Cinder containers: volume and env customizations https://review.openstack.org/533311 | 21:53 |
| *** abishop has quit IRC | 21:56 | |
| *** ecerquei_ has quit IRC | 21:56 | |
| *** dprince has quit IRC | 22:01 | |
| weshay | haleyb, my env came up | 22:04 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: DEBUG ansible inventory creation https://review.openstack.org/533313 | 22:04 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs032: activate config-download https://review.openstack.org/528761 | 22:05 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Workflow execution blacklist support https://review.openstack.org/533315 | 22:06 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Correct panko auth region https://review.openstack.org/532980 | 22:06 |
| *** trown is now known as trown|outtypewww | 22:07 | |
| *** fragatina has joined #tripleo | 22:08 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742936 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 22:10 |
| *** itlinux has quit IRC | 22:10 | |
| *** paramite has joined #tripleo | 22:11 | |
| *** etingof has quit IRC | 22:13 | |
| *** morazi has quit IRC | 22:14 | |
| *** lblanchard has quit IRC | 22:17 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Workflow execution blacklist support https://review.openstack.org/533315 | 22:19 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Also pass blacklisted hostnames https://review.openstack.org/533319 | 22:19 |
| *** fragatina has quit IRC | 22:28 | |
| *** fragatina has joined #tripleo | 22:28 | |
| *** weshay is now known as weshay_PTO | 22:29 | |
| *** jlabarre has quit IRC | 22:32 | |
| *** fragatina has quit IRC | 22:33 | |
| *** toure is now known as toure|gone | 22:33 | |
| *** jkilpatr has joined #tripleo | 22:34 | |
| *** fragatina has joined #tripleo | 22:35 | |
| *** vpickard is now known as vpickard_ | 22:36 | |
| *** eck` is now known as eck`gone | 22:38 | |
| *** eck`gone is now known as eck` | 22:38 | |
| *** dsneddon_afk is now known as dsneddon | 22:41 | |
| *** holser__ has quit IRC | 22:44 | |
| *** holser__ has joined #tripleo | 22:46 | |
| *** jmelvin has quit IRC | 22:49 | |
| *** etingof has joined #tripleo | 22:55 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Use docker_config_scripts for puppet apply https://review.openstack.org/533062 | 22:55 |
| *** raildo has quit IRC | 23:04 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1742936 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 23:10 |
| *** owalsh has quit IRC | 23:12 | |
| *** holser__ has quit IRC | 23:12 | |
| *** owalsh has joined #tripleo | 23:12 | |
| *** lblanchard has joined #tripleo | 23:14 | |
| *** itlinux has joined #tripleo | 23:21 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Workflow execution blacklist support https://review.openstack.org/533315 | 23:23 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Also pass blacklisted hostnames https://review.openstack.org/533319 | 23:23 |
| *** matbu has quit IRC | 23:26 | |
| *** lblanchard has quit IRC | 23:28 | |
| *** etingof has quit IRC | 23:28 | |
| *** etingof has joined #tripleo | 23:29 | |
| *** dougbtv has joined #tripleo | 23:32 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates stable/pike: Fix path for iscsi config file https://review.openstack.org/532523 | 23:36 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates stable/pike: Fix puppet config volume for iscsid in containers https://review.openstack.org/531680 | 23:36 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Revert "Re-enable voting on scenario002-container" https://review.openstack.org/533213 | 23:38 |
| *** itlinux has quit IRC | 23:39 | |
| *** ecerquei has joined #tripleo | 23:52 | |
| *** ecerquei has quit IRC | 23:56 | |
| *** ecerquei has joined #tripleo | 23:57 | |
| *** itlinux has joined #tripleo | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!