| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 00:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 00:10 |
| *** zaneb has joined #tripleo | 00:18 | |
| *** rlandy has quit IRC | 00:58 | |
| *** phuongnh has joined #tripleo | 01:02 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 01:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 01:10 |
| *** ansmith has quit IRC | 01:14 | |
| *** mschuppert has quit IRC | 01:36 | |
| *** lblanchard has quit IRC | 01:51 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 02:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 02:10 |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 03:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 03:10 |
| *** jaganathan has joined #tripleo | 03:16 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 04:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 04:10 |
| *** Petersingh has joined #tripleo | 04:34 | |
| *** ramishra has joined #tripleo | 04:39 | |
| *** Petersingh is now known as Petersingh|afk | 04:41 | |
| *** janki has joined #tripleo | 04:54 | |
| *** shyamb has joined #tripleo | 04:59 | |
| *** shyamb has quit IRC | 05:05 | |
| *** apetrich has joined #tripleo | 05:06 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 05:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 05:10 |
| *** jtomasek has joined #tripleo | 05:17 | |
| *** Petersingh|afk is now known as Petersingh | 05:18 | |
| *** shyamb has joined #tripleo | 05:20 | |
| *** chem has quit IRC | 05:29 | |
| *** yprokule has joined #tripleo | 05:30 | |
| *** jfrancoa has joined #tripleo | 05:33 | |
| *** quiquell|off is now known as quiquell | 05:46 | |
| quiquell | Good morning | 05:46 |
| *** khyr0n has quit IRC | 05:49 | |
| *** ksambor has joined #tripleo | 05:53 | |
| *** ratailor has joined #tripleo | 05:53 | |
| *** aufi_ has joined #tripleo | 05:54 | |
| Tengu | hello there | 05:55 |
| Tengu | quiquell: «o/ | 05:55 |
| quiquell | Tengu: Humm maybe you know something abouit overcloud-ssl https://logs.rdoproject.org/45/560445/160/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/7e81c38/job-output.txt.gz#_2018-10-08_01_15_52_827996 | 05:57 |
| *** yprokule has quit IRC | 05:58 | |
| quiquell | marios: https://logs.rdoproject.org/45/560445/160/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/7e81c38/job-output.txt.gz#_2018-10-08_01_15_52_827996 | 05:58 |
| Tengu | 1s, starting all my things :). I'm a bit late today | 05:58 |
| Tengu | hmm. | 05:59 |
| Tengu | I did stuff on that field indeed. | 05:59 |
| quiquell | Tengu: ack, me too, removing my brain, and putting a chip now. | 05:59 |
| Tengu | but it was one of the first things I did back in May. so I'm a bit surprised it crashes now. | 05:59 |
| Tengu | lemme check | 05:59 |
| quiquell | Tengu: where can I look to help ¿ | 05:59 |
| quiquell | Tengu: I am always a little lost with puppet | 06:00 |
| quiquell | Hum it's not puppet | 06:00 |
| Tengu | nah | 06:00 |
| Tengu | that's some heat stuff. done mainly by jaosorio | 06:00 |
| Tengu | +r | 06:00 |
| Tengu | and it should not be used anymore I think. | 06:00 |
| quiquell | Let's see latest changes | 06:00 |
| quiquell | Tengu: we are rewritting promotion jobs, so maybe the fix is not in th emix | 06:01 |
| Tengu | 11th of June | 06:01 |
| * quiquell like the phras is like a song, "maybe the fix is not in the mix" | 06:01 | |
| Tengu | yeah, trouble is, I spent a couple of hours moving that tls stuff to pure ansible. | 06:02 |
| *** shyamb has quit IRC | 06:02 | |
| Tengu | so I think there's an issue if it's called. Not 100% sure though. | 06:02 |
| quiquell | Tengu: Let's check hashes | 06:02 |
| Tengu | more over... apparently nothing includes nor point to it according to a "git grep tls-cert-inject" | 06:03 |
| Tengu | so I'm surprised. | 06:03 |
| Tengu | where is it called from? | 06:03 |
| quiquell | Tengu: openstack-tripleo-heat-templates-9.0.1-0.20181008002723.061e033.el7.noarch | 06:04 |
| *** shyamb has joined #tripleo | 06:05 | |
| quiquell | Tengu: Humm i see two different hashes at repo setup... let me check that first | 06:07 |
| Tengu | ok :) | 06:07 |
| *** pdeore has joined #tripleo | 06:09 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 06:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 06:10 |
| quiquell | Humm waht is dlrn_hash_tag_newest, | 06:10 |
| quiquell | We have rdl_dlrn 6adc2f3f85c63119d474c6ae8932ae4b154696bd_2a5797bc | 06:13 |
| quiquell | And tripleo_dlrn 965941f1e62cef16967e7a7cd6d98263e52acb62_0989b280 | 06:13 |
| quiquell | We the tripleo-openstack packages from current | 06:16 |
| *** mschuppert has joined #tripleo | 06:21 | |
| quiquell | Ahh ok make sense, we test the tripleo stuff with the latest changes using current, what we promote are the other projects | 06:21 |
| quiquell | Tengu: It's latest tht | 06:21 |
| *** agurenko has joined #tripleo | 06:23 | |
| quiquell | Tengu: Oct 7 was ok | 06:23 |
| Tengu | hmm | 06:24 |
| Tengu | so something was merged in the meantime and creates an issue right? | 06:24 |
| quiquell | Tengu: good one from Oct 7 https://logs.rdoproject.org/45/560445/159/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/bda64de/ | 06:24 |
| Tengu | git bisect to the rescue? | 06:24 |
| *** iurygregory has joined #tripleo | 06:24 | |
| quiquell | Tengu: dlrn from good 2dea64def5045cb4420daaab16ba36e71ec16c9a_24ce4662 | 06:25 |
| quiquell | Tengu: This is the only chagne from tht https://github.com/openstack/tripleo-heat-templates/commit/6adc2f3f85c63119d474c6ae8932ae4b154696bd | 06:27 |
| Tengu | hm. not related I'd say. | 06:27 |
| quiquell | Tengu: yap Can be any of this projects | 06:28 |
| quiquell | > - ansible-role-container-registry | 06:28 |
| quiquell | - ansible-role-tripleo* | 06:28 |
| quiquell | - ansible-tripleo-ipsec | 06:28 |
| quiquell | - instack | 06:28 |
| quiquell | - instack-undercloud | 06:28 |
| quiquell | - openstack-tripleo-* | 06:28 |
| quiquell | - os-apply-config | 06:28 |
| quiquell | - os-collect-config | 06:28 |
| quiquell | - os-net-config | 06:28 |
| quiquell | - os-refresh-config | 06:28 |
| quiquell | - puppet-* | 06:28 |
| quiquell | - python*-tripleo* | 06:28 |
| quiquell | - python*-paunch* | 06:28 |
| Tengu | hmm. is there a way to know where is the "overcloud-ssl : fetch template from single remote host" task defined? isn't it in quickstart* ? | 06:28 |
| *** ratailor has quit IRC | 06:28 | |
| Tengu | I think that's the right thing to look for | 06:29 |
| quiquell | Tengu: http://codesearch.openstack.org/?q=fetch%20template%20from%20single%20remote%20host&i=nope&files=&repos= | 06:29 |
| quiquell | tqe | 06:29 |
| quiquell | Tengu: maybe this ? https://github.com/openstack/tripleo-quickstart-extras/commit/156d14e573c60d897083904e5fefcd460ee418e7 | 06:30 |
| quiquell | tqe is not a DLRN | 06:30 |
| Tengu | so it's the tls_tht "module" apparently. | 06:30 |
| Tengu | ah, that might be more than related indeed. | 06:31 |
| *** mrsoul has joined #tripleo | 06:31 | |
| quiquell | https://github.com/openstack/tripleo-quickstart-extras/commit/bf7a2e22df0ff6c21b6621a1e1cf6fffdca6f711 | 06:32 |
| marios | quiquell: o/ morning folks whats broken this time :/ | 06:32 |
| Tengu | hello marios :) | 06:32 |
| quiquell | marios: We think this has broke the gates https://review.openstack.org/#/c/602171/ | 06:32 |
| Tengu | quiquell: we might want to was for jaosorior :/ | 06:33 |
| quiquell | RDO | 06:33 |
| *** pvc has quit IRC | 06:33 | |
| quiquell | yep ovb fs001 master is broken in the review :-( | 06:33 |
| marios | quiquell: man, there was that undercloud issue on friday, and then it was fixed. i checked grafana yesterday was like 90% pass i was so happy. | 06:33 |
| marios | then monday came | 06:33 |
| Tengu | :) | 06:33 |
| quiquell | marios: Yep... at ruck/rover good is weird, bad is the standard | 06:34 |
| Tengu | http://imgc.allpostersimages.com/images/P-473-488-90/59/5996/W5OQG00Z/posters/i-hate-mondays.jpg | 06:34 |
| marios | quiquell: Tengu so this affects promotions right? I mean all check/gate (voting) are green on that 602171 | 06:34 |
| marios | legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master | 06:34 |
| Tengu | marios: seems to break the RDO mainly | 06:35 |
| marios | Tengu: yeah sorry isn't a promotion job | 06:35 |
| Tengu | but I didn't explore more | 06:35 |
| *** ratailor has joined #tripleo | 06:35 | |
| quiquell | marios: OVB jobs are affected, so promotion is affected (now stopped because we are rewritting it) | 06:36 |
| quiquell | marios: It's clear a promotion-blocker | 06:36 |
| quiquell | marios: no alert needed upstream gates are ok | 06:37 |
| marios | quiquell: ack | 06:37 |
| quiquell | marios: Go back in a few | 06:37 |
| *** quiquell is now known as quiquell|brb | 06:37 | |
| marios | quiquell: looking at http://cistatus.tripleo.org/ yah looks like that starting failing yesterday (some successes before that) | 06:37 |
| marios | thanks Tengu and quiquell|brb | 06:38 |
| *** f2 has joined #tripleo | 06:49 | |
| *** f2 is now known as floriand | 06:49 | |
| *** floriand is now known as florianf | 06:49 | |
| bandini | marios: is there a bug for the ssl thing? https://logs.rdoproject.org/45/560445/160/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/7e81c38/job-output.txt.gz#_2018-10-08_01_15_52_827996 I will start looking at it | 06:50 |
| EmilienM | marios: hey, just fyi over the weekend the patch to switch docker to podman on containerized undercloud job merged | 06:50 |
| EmilienM | marios and all: let me know if there is any trouble | 06:50 |
| marios | EmilienM: ack thanks | 06:50 |
| EmilienM | you can also ping Tengu if I don't respon (I'm in Brno for mtgs) | 06:51 |
| marios | ack | 06:51 |
| * Tengu can try, but doesn't have +2 votes for now ;) | 06:51 | |
| EmilienM | Tengu: what I'm saying is here things go south with podman, we're here to make support | 06:51 |
| EmilienM | we now have it in the gate for one job | 06:52 |
| Tengu | yep | 06:52 |
| Tengu | \o/ | 06:52 |
| EmilienM | marios, bandini: if you can look super quick - https://review.openstack.org/#/c/608456/ - thanks | 06:52 |
| *** cylopez has joined #tripleo | 06:54 | |
| bandini | marios, quiquell|brb: https://review.openstack.org/608589 probably (re oooq and overcloud_ssl error) ? | 06:58 |
| bandini | EmilienM: looking | 06:58 |
| EmilienM | bandini, marios: thank you | 07:00 |
| marios | thanks bandini will check in a bit | 07:01 |
| *** kopecmartin|off is now known as kopecmartin|ruck | 07:02 | |
| hjensas | bandini: quiquell|brb: https://review.openstack.org/608588 <- for the same overcloud_ssl issue. | 07:06 |
| hjensas | marios: ^^ | 07:06 |
| marios | ack hjensas thanks | 07:06 |
| bandini | hjensas: ack. so we do not need OS::TripleO::NodeTLSData at all any longer? | 07:07 |
| hjensas | bandini: marios: I think we should drop mine, I forget that quickstart need to be backward compatible. | 07:07 |
| hjensas | bandini: in master I don't see that resource used anywhere ... | 07:08 |
| bandini | hjensas: I see, I'll abandon mine and point to yours and we can discuss on your review how to best do things. ok? | 07:09 |
| hjensas | bandini: marios: I abondoned mine. | 07:09 |
| bandini | lol | 07:09 |
| hjensas | bandini: I think your's is better. | 07:09 |
| bandini | I have no clue about this tls stuff | 07:09 |
| *** pvc has joined #tripleo | 07:09 | |
| *** quiquell|brb is now known as quiquell | 07:09 | |
| pvc | hi can i disable ipv6 on overcloud node? | 07:09 |
| hjensas | bandini: me neither, just trying to help get as much CI as possible. | 07:09 |
| bandini | same here :) | 07:10 |
| bandini | I'll add the bug to the review | 07:10 |
| marios | thanks hjensas i just went to file the bug kopecmartin|ruck fyi discussion here is about https://bugs.launchpad.net/tripleo/+bug/1796626 | 07:10 |
| openstack | Launchpad bug 1796626 in tripleo "OVB - overcloud-ssl fail with KeyError" [High,In progress] - Assigned to Harald Jensås (harald-jensas) | 07:10 |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 07:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 07:10 |
| quiquell | cool thanks bandini and hjensas | 07:10 |
| *** slaweq_ has joined #tripleo | 07:12 | |
| quiquell | bandini: Why it only affects OVB fs001 at master ? | 07:13 |
| *** tosky has joined #tripleo | 07:13 | |
| *** mburned_out is now known as mburned | 07:14 | |
| bandini | quiquell: can't say as I don't really know my way around ovb/ scenarios etc. | 07:14 |
| quiquell | bandini: When is overcloud-ssl activated ? | 07:15 |
| marios | quiquell: also 35 | 07:16 |
| *** jtomasek has quit IRC | 07:16 | |
| marios | in fact the trace hjensas has in his bug report is for 35 | 07:16 |
| *** jtomasek has joined #tripleo | 07:17 | |
| marios | quiquell: i guess the answer is because ssl_overcloud_true in the featurset but then 2 should also fail /me check | 07:18 |
| marios | quiquell: ah no ovb job for 2 | 07:19 |
| marios | bandini: so are we going with your one right? hjensas ? | 07:19 |
| *** rcernin has quit IRC | 07:21 | |
| bandini | I think so | 07:21 |
| marios | bandini: but i don't get it since environments/enable-internal-tls.yaml environments/ssl/enable-internal-tls.yaml both have resource_registry defined | 07:22 |
| marios | bandini: sorry i'll comment on the review will be easier to discuss there thanks | 07:22 |
| marios | brb | 07:22 |
| bandini | hjensas: I captured your tls removal in mine as well now (I do it for rocky and onwards) | 07:22 |
| pvc | anyone familiar why overcloud node have an error Check cable when booting up | 07:25 |
| *** bogdando has joined #tripleo | 07:29 | |
| hjensas | bandini: thank, I -1'it and put some comments. | 07:31 |
| pvc | marios | 07:35 |
| pvc | bandini | 07:35 |
| pvc | ? | 07:35 |
| pvc | anyone? | 07:35 |
| *** rdopiera has joined #tripleo | 07:37 | |
| *** chandankumar has joined #tripleo | 07:40 | |
| quiquell | bandini: Did it affects queens and pike ? | 07:41 |
| bandini | not sure | 07:42 |
| *** Petersingh is now known as Petersingh|lunch | 07:42 | |
| *** jpena|off is now known as jpena | 07:44 | |
| *** mwhahaha changes topic to "Welcome to Stein | CI Status: GREENish - don't mess this up | https://docs.openstack.org/tripleo-docs/latest/" | 07:44 | |
| *** morazi has joined #tripleo | 07:46 | |
| *** shyamb has quit IRC | 07:48 | |
| hjensas | pvc: sounds like something the NIC might write out in case PXE fails? | 07:48 |
| pvc | hjensas it is on overcloud deploy | 07:50 |
| pvc | hjensas one of my baremetal overcloud server got an IP but the other one fail to disconver on neturon dhcp | 07:50 |
| hjensas | pvc: so you can introspect both? But you overcloud deploy fail's for one? | 07:51 |
| pvc | Yes i can introspect both hjensan and it states become available | 07:52 |
| pvc | but when doing overcloud deploy it failing on firstboot since it doesnt have an IP address to ping | 07:52 |
| hjensas | pvc: where do you see the error? On the baremetal console? | 07:52 |
| *** skramaja has joined #tripleo | 07:52 | |
| pvc | yes on IPMI console if login then no ip address on interface but the nova list said its running, also on neutron logs there is no dhcp discover on that mac address | 07:53 |
| Tengu | fscking bluejean interface changing selected date when you validate a meeting X(. damn app. | 07:59 |
| hjensas | pvc: strange, does the mac of the nic it's trying to DHCP match the mac on the neutron port on the undercloud? | 07:59 |
| pvc | yes it is strange because the other overcloud succesfully map the IP address given by neutron to its mac address | 08:00 |
| *** skramaja_ has joined #tripleo | 08:02 | |
| *** skramaja has quit IRC | 08:02 | |
| *** owalsh_ is now known as owalsh | 08:03 | |
| hjensas | pvc: if you do openstack baremetal node port list --node <uuid-of-ironic-node>, And then openstack baremetal port show <uuid's returned by previous command> . Does the MAC address on the ironic port's match the mac you see on the console? | 08:04 |
| *** amoralej|off is now known as amoralej | 08:04 | |
| pvc | Noted on this wait hjensas | 08:06 |
| *** jpich has joined #tripleo | 08:07 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 08:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 08:10 |
| *** dciabrin has joined #tripleo | 08:10 | |
| pvc | hjensan yes it matches | 08:15 |
| pvc | mac address of the node list versus the mac address of the console | 08:16 |
| pvc | it is possible that i will just run the command "ifup <interface name>" to make it pingable hjensas? | 08:21 |
| hjensas | pvc: I would run the dhcp client, and then use debug on switches and tcpdump to try to figure out where the dhcp requests are dropped. | 08:26 |
| hjensas | pvc: you can try to use ifup, and see if you can ping it if you set an address. | 08:27 |
| pvc | i can run ifup and it can get an ip address, ip address on the nova list | 08:27 |
| *** fhubik has joined #tripleo | 08:27 | |
| pvc | we are using vlan hjensas | 08:27 |
| pvc | is that possible that i will just ifup the interface name? Is my deployment will continue? What is the first step overcloud deploy will after spawning and running the instance hjensas | 08:30 |
| hjensas | pvc: ok, could it be spanning-tree taking a lot of time to converge the port to a forwarding state when the link comes up? | 08:30 |
| pvc | i really dont know :(. If the two server is pingable it is okay now? | 08:31 |
| pvc | how can i check that hjensas? | 08:32 |
| hjensas | pvc: it depends, check log's (journal) if os-collect-config is doing something? If not, try to restart os-collect-config ? | 08:32 |
| pvc | hjensas journalctl -u os-collect-config | 08:33 |
| hjensas | pvc: yes | 08:33 |
| *** paramite has joined #tripleo | 08:34 | |
| pvc | okay noted on this. thank you. i will just wait the instnace to boot hjensa | 08:34 |
| *** shardy has joined #tripleo | 08:39 | |
| *** moguimar has joined #tripleo | 08:44 | |
| *** chem has joined #tripleo | 08:45 | |
| *** Petersingh|lunch is now known as Petersingh | 08:47 | |
| *** chandankumar has quit IRC | 08:51 | |
| *** shyamb has joined #tripleo | 08:55 | |
| *** jrist has joined #tripleo | 08:57 | |
| *** jrist has quit IRC | 09:02 | |
| pvc | hjensas it can succesfuly dpeloy ramdisk and kernel but on the next reboot it cannot up the interface, i will just run the command "ifup <interface name> | 09:02 |
| *** salmankhan has joined #tripleo | 09:05 | |
| pvc | hjensan i already run the command "ifup <interface name" and it can get an ip address | 09:09 |
| *** aufi has joined #tripleo | 09:09 | |
| pvc | hjensas i already run the command "ifup <interface name" and it can get an ip address. | 09:09 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 09:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 09:10 |
| pvc | hjensas -- No entries -- when running journalctl -u os-collect-config | 09:11 |
| *** aufi_ has quit IRC | 09:12 | |
| janki | Hi. Can I get some reviews for a cherry-pick plz https://review.openstack.org/#/c/608564/ | 09:12 |
| *** sri_ has joined #tripleo | 09:12 | |
| *** yprokule has joined #tripleo | 09:20 | |
| pvc | hjensas i manually run the os-collect-config | 09:23 |
| pvc | hi anyone can check this? http://paste.openstack.org/show/731672/ | 09:26 |
| *** pvc_ has joined #tripleo | 09:28 | |
| pvc_ | hjensas for compute node http://paste.openstack.org/show/731673/ | 09:28 |
| *** pvc has quit IRC | 09:30 | |
| hjensas | pvc_: I am not sure what, but most likely some configuration that would normally be loaded over the network is missing dues to the initially issue forcing you to manually ifup the interface. | 09:35 |
| pvc_ | it is okay now i just manually up the interface then run the os-collect-config | 09:36 |
| pvc_ | but its failing on some tasks here http://paste.openstack.org/show/731673/ ( compute node ) | 09:37 |
| hjensas | pvc_: I belive that depracation message is normal. It's INFO level message, not an error. | 09:38 |
| *** sshnaidm is now known as sshnaidm|lnch | 09:40 | |
| *** shyamb has quit IRC | 09:41 | |
| pvc_ | this one error on cinder.pp hjensas http://paste.openstack.org/show/731676/ | 09:41 |
| *** agurenko has quit IRC | 09:41 | |
| *** agurenko has joined #tripleo | 09:43 | |
| *** shyamb has joined #tripleo | 09:43 | |
| *** dtantsur|afk is now known as dtantsur | 09:49 | |
| hjensas | pvc_: oh, I'm not sure where the code that should write that hiereadata lives. But I would go back and figure out why your network does'nt come up as it should, this is likely the reason things don't happen the way they should. | 09:50 |
| *** jfrancoa has quit IRC | 09:52 | |
| Tengu | hey guys, anyway to get a w+1 on that one? https://review.openstack.org/#/c/583106/ | 09:53 |
| *** jfrancoa has joined #tripleo | 09:57 | |
| skramaja_ | shardy: could you take a look https://review.openstack.org/#/c/597988/ and https://review.openstack.org/#/c/598052/? | 09:58 |
| *** sshnaidm|lnch is now known as sshnaidm | 09:59 | |
| *** iurygregory is now known as iurygregory|lunc | 09:59 | |
| shardy | skramaja_: ack sure will do | 10:01 |
| pdeore | Hi everyone ! Can someone review this patch please? It's kinda blocker in OSP14 https://review.openstack.org/#/c/598560/ | 10:01 |
| skramaja_ | thanks shardy | 10:01 |
| shardy | "CI Status: GREENish - don't mess this up" lol :D | 10:02 |
| skramaja_ | :) | 10:02 |
| dtantsur | morning! if the CI is finally greenish, can someone please merge https://review.openstack.org/#/c/601621/ ? :) | 10:06 |
| *** jpena is now known as jpena|off | 10:07 | |
| hjensas | bogdando: Hey, regarding logging. Why is'nt all services logging into /var/log/container/ ? For example: https://github.com/openstack/tripleo-heat-templates/blame/6adc2f3f85c63119d474c6ae8932ae4b154696bd/docker/services/novajoin.yaml#L167-L171 | 10:07 |
| bogdando | hjensas: from kolla pov, its /var/log | 10:08 |
| bogdando | the latter bind mounted into host /var/log/container | 10:08 |
| bogdando | ah, that /dev/stdout. Perhaps a bug, hjensas | 10:09 |
| *** jtomasek has quit IRC | 10:09 | |
| hjensas | bogdando: yep, I think I get the bind mount stuff, but novajoin uses stdout which means I see some things in journal but no logfile in /var/log/container . | 10:10 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 10:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 10:10 |
| bogdando | hjensas: a bug, indeed | 10:10 |
| *** janki has quit IRC | 10:12 | |
| *** janki has joined #tripleo | 10:13 | |
| hjensas | bogdando: we should just drop '--log-file /dev/stdout' ? I see config files have 'log_dir=/var/log/novajoin' | 10:13 |
| bogdando | hjensas, owalsh: hi, around? WDYT? ^^ | 10:20 |
| *** quiquell is now known as quiquell|brb | 10:20 | |
| bogdando | perhaps we should just drop it. And see how CI works | 10:20 |
| *** quiquell|brb is now known as quiquell | 10:21 | |
| owalsh | bogdando: hey, not sure. novajoin has nothing to do with nova.... jaosorior is the guy to talk to | 10:22 |
| *** ssbarnea_ has quit IRC | 10:24 | |
| *** aufi_ has joined #tripleo | 10:25 | |
| pvc_ | hi anyone encounter this? | 10:25 |
| pvc_ | Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/cinder.pp:87:30 | 10:25 |
| *** jaosorior has joined #tripleo | 10:26 | |
| *** aufi has quit IRC | 10:27 | |
| *** pvc has joined #tripleo | 10:28 | |
| pvc | Error: Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/cinder.pp:87:30 | 10:28 |
| *** jtomasek has joined #tripleo | 10:28 | |
| *** shyamb has quit IRC | 10:29 | |
| *** pvc_ has quit IRC | 10:29 | |
| *** pdeore has quit IRC | 10:33 | |
| hjensas | bogdando: I filed a bug. https://bugs.launchpad.net/tripleo/+bug/1796658 <- jaosorior any idea why novajoin log's is done different than other services? | 10:34 |
| openstack | Launchpad bug 1796658 in tripleo "novajoin logging not correct in container undercloud" [Medium,Triaged] | 10:34 |
| jaosorior | hjensas: it was done with the idea that it's the "container" way of doing things. As opposed to logging to files which is not a good pattern. Given the low adoption of novajoin, it seemed safe to do so, but docs are needed | 10:35 |
| *** rcernin has joined #tripleo | 10:36 | |
| jaosorior | hjensas: docker logs, or journalctl CONTAINER_NAME=<novajoin container> would be the right way get the logs | 10:36 |
| hjensas | jaosorior: ack, I don't disagree with the reasoning. But it's not consistent with other services, which imo is also not good. | 10:37 |
| *** Petersingh is now known as Petersingh|afk | 10:41 | |
| *** gvrangan has joined #tripleo | 10:43 | |
| *** slaweq_ is now known as slaweq | 10:44 | |
| *** slaweq is now known as dpawlik_ | 10:45 | |
| *** dpawlik_ is now known as slaweq | 10:45 | |
| *** aufi has joined #tripleo | 10:46 | |
| *** aufi_ has quit IRC | 10:48 | |
| *** fhubik has left #tripleo | 10:51 | |
| *** quiquell is now known as quiquell|brb | 10:53 | |
| *** aedc has joined #tripleo | 10:57 | |
| *** boazel has quit IRC | 10:58 | |
| pvc | hjensan | 11:03 |
| pvc | hjensas | 11:03 |
| pvc | how can i assign a public subnet for nic2 to be used by br-ex? hjensas | 11:03 |
| *** aufi_ has joined #tripleo | 11:04 | |
| *** shyamb has joined #tripleo | 11:05 | |
| *** aufi has quit IRC | 11:06 | |
| *** gvrangan has quit IRC | 11:08 | |
| *** phuongnh has quit IRC | 11:09 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 11:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 11:10 |
| Tengu | hey all, PTAL? https://review.openstack.org/#/c/605450/ thank you | 11:11 |
| pvc | Tengu do i need to manually edit this /etc/os-collect-config.conf/ | 11:13 |
| *** psachin has joined #tripleo | 11:13 | |
| Tengu | err... no context - but I think it's not a good idea as its content will more than probably be overidden at some point | 11:13 |
| hjensas | pvc: I'm not sure I follow the question. You probably want to enable network-isolation. You configure the networks in network_data.yaml and roles_data.yaml. | 11:15 |
| pvc | for this hjensas /etc/os-collect-config.conf/? | 11:15 |
| pvc | should i edit this on undercloud? | 11:15 |
| hjensas | no, that's not what you want to edit. | 11:16 |
| hjensas | pvc: https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/custom_networks.html | 11:17 |
| *** rcernin has quit IRC | 11:17 | |
| hjensas | pvc: If you don't use network isolation, you can make a stripped down version of the defaults adding only the external network. | 11:18 |
| pvc | i dont have this on ocata /usr/share/openstack-tripleo-heat-templates/network_data.yaml | 11:18 |
| pvc | what about this hjensas: os-collect-config on each deployed server must be manually configured to poll the Heat API for the available SoftwareDeployments. An example configuration for /etc/os-collect-config.conf looks like: | 11:18 |
| *** ratailor has quit IRC | 11:20 | |
| hjensas | pvc: that's if you use deployed-server - https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/deployed_server.html | 11:20 |
| pvc | may baremetal node have a CentoS server | 11:21 |
| *** boazel has joined #tripleo | 11:24 | |
| pvc | what will i run in order to successfully finished the NetworkDeployment hjensan? | 11:25 |
| pvc | what will i run in order to successfully finished the NetworkDeployment hjensas? | 11:25 |
| pvc | so i dont need to run the os-collect-config? | 11:29 |
| *** jamesdenton has quit IRC | 11:30 | |
| *** Petersingh|afk is now known as Petersingh | 11:32 | |
| *** iurygregory|lunc is now known as iurygregory | 11:32 | |
| Tengu | jaosorior: heya! care to have a look at the patches under https://review.openstack.org/#/q/topic:podman/standalone-deploy+(status:open+OR+status:merged) ? they need some love in order to get merged for podman/selinux to work :D | 11:39 |
| pvc | os-collect-config is only for deployed servers hjensas? | 11:46 |
| hjensas | pvc: no os-collect-config is always used. But you only need to manually configure it if you are using deployed servers. | 11:46 |
| pvc | im not using deployed server templates but my overcloud node have an centos OS on it hjensas | 11:47 |
| pvc | should i wait on NetworkDeployment process to finish? Even if i run the command "ifup <interface name>" to make the overcloud to be reachable | 11:48 |
| hjensas | pvc: if your overcloud servers have a pre-deployed CentOS, and you don't intend to re-image the servers using TripleO you have to use deployed-server templates. | 11:49 |
| *** morazi has quit IRC | 11:49 | |
| pvc | i see i will just use the re-image using TripleO. but what is the next step should i monitor if my 2 overcloud nodes is reacheable on undercloud? | 11:50 |
| *** jrist has joined #tripleo | 11:50 | |
| jaosorior | Tengu: will do | 11:50 |
| Tengu | jaosorior: thanks :) | 11:51 |
| pvc | should i need to wait 40mins hjensas? | 11:52 |
| pvc | because it's not creating the br-ex for the controller node even if i up the interface | 11:53 |
| *** jrist has quit IRC | 11:54 | |
| hjensas | pvc: If it's been on NetworkDeployment for 40 minutes it's likely that the network configuration on the overcloud node is incorrect, causing the overcloud node to be unable to signal back to the undercloud that it failed. If that is the case, the deployment will eventually timeout and error. | 11:55 |
| pvc | but if i run the os-collect-config manually it can create a br-ex interface on the controller node but that is not the right way since i didnt use the deployed server tempaltes | 11:56 |
| *** shyamb has quit IRC | 12:01 | |
| *** shyamb has joined #tripleo | 12:01 | |
| *** raildo has joined #tripleo | 12:02 | |
| *** rlandy has joined #tripleo | 12:04 | |
| *** rh-jelabarre has joined #tripleo | 12:08 | |
| *** leanderthal has joined #tripleo | 12:08 | |
| Tengu | quiquell|brb: did you ping jaosorior about the TLS issue on the CI? | 12:09 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 12:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 12:10 |
| Tengu | ah, yep, apparently seeing the correction :). | 12:10 |
| Tengu | cool. | 12:10 |
| jaosorior | ?? | 12:10 |
| Tengu | jaosorior: related to https://review.openstack.org/#/c/608589/ | 12:11 |
| jaosorior | oh, right | 12:11 |
| Tengu | :) | 12:12 |
| Tengu | ah that one would need some review and w+1 I think - it will unlock a current bad situation: https://review.openstack.org/#/c/607491/ | 12:13 |
| jaosorior | marios: could you check this out https://review.openstack.org/#/c/607953/ ? | 12:14 |
| *** dprince has joined #tripleo | 12:15 | |
| *** trown|outtypewww is now known as trown | 12:18 | |
| *** weshay_pto is now known as weshay | 12:19 | |
| *** jamesdenton has joined #tripleo | 12:20 | |
| *** quiquell|brb is now known as quiquell | 12:20 | |
| quiquell | Tengu: we have a correct ion for it | 12:21 |
| *** aufi_ has quit IRC | 12:23 | |
| *** bfournie has joined #tripleo | 12:23 | |
| Tengu | quiquell: saw that, good | 12:24 |
| *** mrunge_ has quit IRC | 12:25 | |
| *** mrunge has joined #tripleo | 12:25 | |
| *** lblanchard has joined #tripleo | 12:35 | |
| jaosorior | hjensas: around? | 12:46 |
| *** Petersingh is now known as Petersingh|gone | 12:48 | |
| *** Petersingh|gone has quit IRC | 12:48 | |
| *** tzumainn has joined #tripleo | 12:49 | |
| *** shyam89 has joined #tripleo | 12:50 | |
| *** shyamb has quit IRC | 12:51 | |
| *** ansmith has joined #tripleo | 12:51 | |
| *** moguimar has quit IRC | 12:51 | |
| *** aufi_ has joined #tripleo | 12:57 | |
| *** psachin has quit IRC | 13:01 | |
| *** morazi has joined #tripleo | 13:04 | |
| *** shyam89 has quit IRC | 13:09 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 13:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 13:10 |
| *** shyam89 has joined #tripleo | 13:11 | |
| *** mjturek has joined #tripleo | 13:12 | |
| *** toure|gone is now known as toure | 13:15 | |
| *** lblanchard has quit IRC | 13:25 | |
| *** zaneb has quit IRC | 13:29 | |
| *** zaneb has joined #tripleo | 13:29 | |
| *** shyam89 has quit IRC | 13:30 | |
| kopecmartin|ruck | marios|rover, fyi, I found this issue (quite old one) in check job , i commented there https://bugs.launchpad.net/tripleo/+bug/1786520 | 13:36 |
| openstack | Launchpad bug 1786520 in tripleo "3node jobs failing due to missing file UpgradeInitDeployment" [Critical,Triaged] | 13:36 |
| *** vinaykns has joined #tripleo | 13:36 | |
| *** agopi is now known as agopi|brb | 13:36 | |
| *** zul has quit IRC | 13:38 | |
| *** jaosorior has quit IRC | 13:39 | |
| trown | mandre: I am trying to work on the idempotency bug, but I am a bit confused by this section https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/services/openshift-master.yaml#L192-L208 in relation to this one https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/services/openshift-master.yaml#L330-L344 | 13:40 |
| *** agopi|brb has quit IRC | 13:41 | |
| trown | mandre: it seems like we would only use openshift-master to scale-up masters, but there is stuff in there for scaling up workers | 13:41 |
| mandre | trown: did you see https://review.openstack.org/#/c/608658/? | 13:41 |
| trown | mandre: nope :P ... you beat me to it | 13:42 |
| mandre | trown: is it the same bug you're talking about? | 13:43 |
| trown | mandre: kind of, it will complete the fix for the idempotency bug as a side effect of your patch in any case | 13:43 |
| trown | mandre: where is "has_new_nodes" defined in your patch though? | 13:44 |
| trown | mandre: oh in the openshift-node service I see it | 13:44 |
| mandre | trown: cool, if there's a related bug to reference in the commit message feel free to modify the patch | 13:44 |
| marios | kopecmartin|ruck ack | 13:45 |
| trown | mandre: yep, I will try that out, and update with the bug if it helps | 13:45 |
| mandre | trown: yeah and the fact is set in an earlier step | 13:45 |
| mandre | trown: it becomes urgent we merge the rest of the patches from https://etherpad.openstack.org/p/tripleo-openshift-patches | 13:48 |
| *** moguimar has joined #tripleo | 13:49 | |
| trown | mandre: ya I can go through the backports since we have good CI right now | 13:50 |
| mandre | trown: thanks! | 13:50 |
| *** janki has quit IRC | 13:51 | |
| *** boazel has quit IRC | 13:51 | |
| *** janki has joined #tripleo | 13:52 | |
| *** boazel has joined #tripleo | 14:00 | |
| *** asbishop has joined #tripleo | 14:02 | |
| *** agopi|brb has joined #tripleo | 14:08 | |
| *** agopi|brb is now known as agopi | 14:09 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 14:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 14:10 |
| *** lblanchard has joined #tripleo | 14:12 | |
| *** SteelyDan is now known as dansmith | 14:13 | |
| *** beekneemech is now known as bnemec | 14:22 | |
| ramishra | marios: Hi, around? | 14:23 |
| *** dtrainor has joined #tripleo | 14:26 | |
| *** bfournie has quit IRC | 14:27 | |
| trown | mandre: I think we need this one https://review.openstack.org/#/c/608687/ for https://review.openstack.org/608665 | 14:27 |
| *** boazel has quit IRC | 14:28 | |
| *** mfedosin has joined #tripleo | 14:33 | |
| marios | ramishra: o/ hi was on calls | 14:34 |
| marios | ramishra: i added you to that review because you landed the check originally | 14:35 |
| mandre | trown: indeed, thanks for the backport | 14:36 |
| marios | ramishra: thanks for the review i'll take a closer look | 14:36 |
| *** jaganathan has quit IRC | 14:37 | |
| ramishra | marios: np:) From the logs it seems little confusing. There seems to be quite a few restarts and it works in a later time, may be something to do with cached policies.. | 14:41 |
| ramishra | marios: updated the bug with my findings. quite late for me, so will have a look tomorrow, if not fixed by then:) | 14:43 |
| marios|rover | thanks ramishra | 14:43 |
| *** lblanchard has quit IRC | 14:43 | |
| *** quiquell is now known as quiquell|off | 14:44 | |
| *** itlinux has quit IRC | 14:50 | |
| *** vinaykns has quit IRC | 14:55 | |
| *** boazel has joined #tripleo | 15:04 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 15:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 15:10 |
| *** agurenko has quit IRC | 15:11 | |
| *** janki has quit IRC | 15:20 | |
| *** panda is now known as panda|off | 15:23 | |
| *** leanderthal has quit IRC | 15:30 | |
| weshay | mwhahaha, fyi.. added alert on https://bugs.launchpad.net/tripleo/+bug/1796626 https://review.openstack.org/#/c/608589/ as it's killing ovb jobs | 15:32 |
| openstack | Launchpad bug 1796626 in tripleo "OVB - overcloud-ssl fail with KeyError - breaks legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master and featureset035" [High,In progress] - Assigned to Michele Baldessari (michele) | 15:32 |
| *** agopi is now known as agopi|lunch | 15:32 | |
| *** aedc has quit IRC | 15:34 | |
| mwhahaha | k | 15:35 |
| *** moguimar has quit IRC | 15:36 | |
| *** moguimar has joined #tripleo | 15:39 | |
| *** yprokule has quit IRC | 15:40 | |
| *** ksambor has quit IRC | 15:41 | |
| *** itlinux has joined #tripleo | 15:43 | |
| *** skramaja_ has quit IRC | 15:45 | |
| weshay | kopecmartin|ruck, ping | 15:47 |
| kopecmartin|ruck | weshay, yes? | 15:47 |
| weshay | kopecmartin|ruck, gate failrue on tempest http://logs.openstack.org/47/607147/7/gate/tripleo-ci-centos-7-undercloud-containers/0aef64e/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-10-08_15_21_04 | 15:47 |
| weshay | kopecmartin|ruck, can you bug that and put in the tempest skip list please | 15:47 |
| kopecmartin|ruck | weshay, i can file a bug, i wonder why I can't see that failure here: http://cistatus.tripleo.org/gates/ | 15:49 |
| bogdando | weshay: I thought you're on PTO! | 15:50 |
| bogdando | please merge https://review.openstack.org/#/c/593103/ | 15:50 |
| bogdando | it is needed by https://review.openstack.org/#/c/576746/ | 15:50 |
| bogdando | weshay ^^ there we go >,< | 15:50 |
| *** sshnaidm is now known as sshnaidm|afk | 15:51 | |
| weshay | kopecmartin|ruck, it just failed | 15:52 |
| weshay | kopecmartin|ruck, I use cistatus when looking at trends, not immediately required data.. it updates once per hour | 15:52 |
| weshay | kopecmartin|ruck, you should see it here soon http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 | 15:53 |
| *** ssbarnea|bkp2 has joined #tripleo | 15:53 | |
| weshay | kopecmartin|ruck, and of course you will find it here http://zuul.openstack.org/ | 15:54 |
| weshay | now | 15:54 |
| kopecmartin|ruck | weshay, ok, the job also failed yesterday with similar error, can it be related? http://logs.openstack.org/51/586251/3/gate/tripleo-ci-centos-7-undercloud-containers/03a2fbd/logs/undercloud/home/zuul/tempest.log.txt.gz | 15:55 |
| *** jfrancoa has quit IRC | 15:55 | |
| weshay | kopecmartin|ruck, ya.. nice | 15:56 |
| weshay | kopecmartin|ruck, so 2018-10-08 15:21:04 | tempest.api.compute.admin.test_aggregates_negative.AggregatesAdminNegativeTestJSON.test_aggregate_add_non_exist_host[id-0ef07828-12b4-45ba-87cc-41425faf5711,negative] | 15:56 |
| weshay | failed twice in the gate plus the other two tests | 15:56 |
| weshay | I would bug them all as inconsistent, or race condition issues | 15:57 |
| weshay | kopecmartin|ruck, as they all passed on the same patch if they failed in gate | 15:57 |
| weshay | must pass in check first to move to gate | 15:57 |
| weshay | kopecmartin|ruck, let's bug the issue, add them to the skip list, and escalate to the nova folks | 15:58 |
| weshay | kopecmartin|ruck, make sense, any questions? | 15:58 |
| weshay | marios|rover, fyi ^ | 15:58 |
| kopecmartin|ruck | weshay, ok, not now, I"ll fill a bug, then I'll ask | 15:58 |
| *** dprince has quit IRC | 16:01 | |
| *** jrist has joined #tripleo | 16:02 | |
| weshay | kopecmartin|ruck, tags for the bug should include alert, promotion-blocker fyi since it's reseting the gate | 16:03 |
| weshay | we'll remove alert once it's in the skip list | 16:03 |
| *** agurenko has joined #tripleo | 16:05 | |
| *** jrist has quit IRC | 16:07 | |
| kopecmartin|ruck | weshay, https://bugs.launchpad.net/tripleo/+bug/1796710 | 16:09 |
| openstack | Launchpad bug 1796710 in tripleo "Tempest tests failed with Read timed out error on tripleo-ci-centos-7-undercloud-containers" [Undecided,New] | 16:09 |
| *** iurygregory is now known as iurygregory|away | 16:09 | |
| weshay | kopecmartin|ruck, k.. status = triage, Importance = critical | 16:09 |
| weshay | kopecmartin|ruck, target milestone = $next_milestone | 16:09 |
| weshay | stien1 | 16:09 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 16:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 16:10 |
| kopecmartin|ruck | weshay, oh, I don't have permissions to set it | 16:11 |
| *** agopi|lunch is now known as agopi | 16:11 | |
| weshay | kopecmartin|ruck, ah k.. I'll do it | 16:11 |
| weshay | kopecmartin|ruck, so next step would be https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/vars/tempest_skip_master.yml | 16:11 |
| weshay | adding to the skip list for master and maybe rocky.. not sure if it failed in rocky | 16:12 |
| * weshay looks | 16:12 | |
| *** jrist has joined #tripleo | 16:12 | |
| weshay | ya.. only see master jobs | 16:13 |
| *** morazi has quit IRC | 16:14 | |
| weshay | kopecmartin|ruck, arxcruz may need help w/ debugging https://bugs.launchpad.net/tripleo/+bug/1796710 | 16:15 |
| openstack | Launchpad bug 1796710 in tripleo "Tempest tests failed with Read timed out error on tripleo-ci-centos-7-undercloud-containers" [Undecided,New] | 16:15 |
| kopecmartin|ruck | weshay, what about the reason of skip, something like "Read timed out" is enough? | 16:16 |
| weshay | kopecmartin|ruck, for now yes, as long as you also have the bug :) | 16:16 |
| weshay | kopecmartin|ruck, you can try to reproduce the error w/ http://logs.openstack.org/47/607147/7/gate/tripleo-ci-centos-7-undercloud-containers/0aef64e/logs/reproducer-quickstart.sh | 16:16 |
| weshay | however since it's a race, your results may vary | 16:16 |
| weshay | thanks man | 16:16 |
| *** jrist has quit IRC | 16:17 | |
| *** mburned is now known as mburned_out | 16:17 | |
| *** roger2 has joined #tripleo | 16:18 | |
| *** bogdando has quit IRC | 16:19 | |
| roger2 | Hello. I'm new here. I've been trying to deploy OpenStack with and without the Quickstart, but haven't completely succeeded yet. Thought I'd start lurking here too and see what I can learn. | 16:19 |
| *** dtantsur is now known as dtantsur|afk | 16:20 | |
| *** rdopiera has quit IRC | 16:21 | |
| *** boazel has quit IRC | 16:21 | |
| *** panda|off has quit IRC | 16:22 | |
| *** panda has joined #tripleo | 16:23 | |
| *** aufi_ has quit IRC | 16:23 | |
| arxcruz | weshay: wel... it's passing now on 607147 | 16:23 |
| weshay | arxcruz, it's inconsistent | 16:23 |
| weshay | arxcruz, needs to be put in skip | 16:23 |
| weshay | until it's 100% | 16:23 |
| kopecmartin|ruck | weshay, https://review.openstack.org/#/c/608723/ I can't add you as a reviewer o.O | 16:24 |
| weshay | ya.. I switched to my redhat email | 16:24 |
| arxcruz | weshay: me neither... check your credentials, i'm also not seeing your gmail... | 16:24 |
| weshay | I'll add myself | 16:24 |
| weshay | add me works | 16:25 |
| weshay | however whayutin@redhat.com | 16:25 |
| weshay | does not | 16:25 |
| weshay | not 100% sure why yet | 16:25 |
| mwhahaha | rascasoft: hi, anything in particular that you're running into? | 16:25 |
| mwhahaha | rascasoft: nevermind, not you :D | 16:26 |
| weshay | arxcruz, kopecmartin|ruck it's fairly consistently working http://zuul.openstack.org/builds.html | 16:28 |
| weshay | job: tripleo-ci-centos-7-undercloud-containers | 16:28 |
| weshay | however not well enough :) | 16:28 |
| kopecmartin|ruck | arxcruz, I saw you patch about stackwiz, .. the failure it is supposed to fix is something like this? https://bugs.launchpad.net/tripleo/+bug/1796040 | 16:30 |
| openstack | Launchpad bug 1796040 in tripleo "Define stestr facts conditaion check fails in validate-tempest role" [Undecided,New] | 16:30 |
| arxcruz | kopecmartin|ruck: yes, it fixes that | 16:30 |
| kopecmartin|ruck | arxcruz, great, just wanted to make sure | 16:30 |
| *** jpich has quit IRC | 16:31 | |
| roger2 | I noticed quickstart.sh defaults to release "queens". Is release "rocky" stable with quickstart.sh? | 16:33 |
| *** moguimar has quit IRC | 16:34 | |
| weshay | roger2, ya.. you should get the same results as https://ci.centos.org/job/tripleo-quickstart-promote-rocky-rdo_trunk-minimal/29/console | 16:35 |
| roger2 | weshay: thank you | 16:35 |
| arxcruz | weshay: do you know which collect-logs rdo jobs execute? i need to unzip stackviz data there, the logs.rdoproject.org doesn't handle zipped files like logs.o.org | 16:36 |
| weshay | arxcruz, in the jobs for ci.centos or other? | 16:37 |
| weshay | rdoproject | 16:37 |
| * weshay looks | 16:37 | |
| arxcruz | weshay: http://logs.rdoproject.org/19/605419/11/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/9508425/ for example | 16:37 |
| weshay | https://logs.rdoproject.org/24/567224/117/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/acd644d/logs/ovb_collect_logs.sh | 16:38 |
| weshay | https://logs.rdoproject.org/24/567224/117/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/acd644d/logs/quickstart_collect_logs.txt.gz | 16:38 |
| weshay | arxcruz, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-rocky/ee6ea96/logs/collect_logs.sh | 16:39 |
| weshay | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-rocky/ee6ea96/logs/quickstart_collect_logs.log | 16:39 |
| weshay | arxcruz, you are probably wondering about the infra part of it though eh? | 16:39 |
| arxcruz | weshay: yup | 16:40 |
| *** ramishra has quit IRC | 16:40 | |
| *** paramite has quit IRC | 16:40 | |
| weshay | arxcruz, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-rocky/ee6ea96/job-output.txt.gz#_2018-10-04_08_10_01_778093 | 16:41 |
| *** paramite has joined #tripleo | 16:41 | |
| arxcruz | weshay: thanks, i think that's all i need, i'll work on a second patch to fix stackviz for rdoproject | 16:41 |
| weshay | thank you! | 16:42 |
| weshay | I'm not sure if that link I sent is correct.. | 16:42 |
| weshay | need to look at the infra playbooks called | 16:42 |
| weshay | arxcruz, note.... | 16:42 |
| weshay | that the rdo jobs are moving to zuulv3 native now | 16:42 |
| weshay | and may zip up everything exactly as upstream does now now too | 16:43 |
| arxcruz | weshay: so, need to add a gunzip in that role | 16:43 |
| weshay | kopecmartin|ruck, need to add Related-Bug: #1796710 to https://review.openstack.org/#/c/608723/ | 16:44 |
| openstack | bug 1796710 in tripleo "Tempest tests failed with Read timed out error on tripleo-ci-centos-7-undercloud-containers" [Undecided,New] https://launchpad.net/bugs/1796710 | 16:44 |
| weshay | kopecmartin|ruck, right above Change-Id: | 16:44 |
| weshay | https://wiki.openstack.org/wiki/GitCommitMessages#Git_Commit_Good_Practice | 16:45 |
| kopecmartin|ruck | weshay, i was thinking about it, but when it doesn't fix the issue, I decided to not to mention it there like that .. ok I'll fix it | 16:45 |
| weshay | kopecmartin|ruck, k.. thanks | 16:45 |
| weshay | ya.. Related-Bug: is good for that | 16:45 |
| weshay | Closes-Bug: #1234567 -- use 'Closes-Bug' if the commit is intended to fully fix and close the bug being referenced. | 16:46 |
| openstack | bug 1234567 in GNU Mailman "Czech catalog bug" [Low,Fix released] https://launchpad.net/bugs/1234567 - Assigned to Mark Sapiro (msapiro) | 16:46 |
| weshay | Partial-Bug: #1234567 -- use 'Partial-Bug' if the commit is only a partial fix and more work is needed. | 16:46 |
| weshay | Related-Bug: #1234567 -- use 'Related-Bug' if the commit is merely related to the referenced bug. | 16:46 |
| kopecmartin|ruck | weshay, ok , thanks, I'll remember that | 16:46 |
| *** sshnaidm|afk is now known as sshnaidm | 16:48 | |
| *** aedc has joined #tripleo | 16:59 | |
| *** trown is now known as trown|lunch | 17:02 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 17:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 17:10 |
| *** apetrich has quit IRC | 17:11 | |
| *** paramite has quit IRC | 17:15 | |
| *** salmankhan has quit IRC | 17:16 | |
| *** cylopez has quit IRC | 17:27 | |
| *** ssbarnea_ has joined #tripleo | 17:29 | |
| *** ssbarnea|bkp2 has quit IRC | 17:30 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:32 | |
| *** shardy has quit IRC | 17:40 | |
| *** apetrich has joined #tripleo | 17:47 | |
| itlinux | hello all on my mac I get this Documents/OpenStack/tripleo-docs/.tox/pep8/bin/flake8' (exited with code 1) | 17:50 |
| itlinux | any tips on how to fix it? | 17:50 |
| *** aedc has quit IRC | 18:00 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 18:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 18:10 |
| *** boazel has joined #tripleo | 18:10 | |
| *** trown|lunch is now known as trown | 18:11 | |
| *** aedc has joined #tripleo | 18:25 | |
| *** panda has quit IRC | 18:26 | |
| *** panda has joined #tripleo | 18:28 | |
| mwhahaha | hjensas, dsneddon_away: so i think something broke in rocky such that if you don't specify a cidr it's setting the <network>_subnet to <ip>/None. Example, "management_subnet": "192.168.24.12/None" | 18:43 |
| mwhahaha | hjensas, dsneddon_away have you seen that before? | 18:43 |
| mwhahaha | this is w/o network isolation so litterally just openstack overcloud deploy --templates (and a handful of unrelated things) | 18:44 |
| *** kopecmartin|ruck is now known as kopecmartin|off | 18:45 | |
| *** slagle has joined #tripleo | 18:52 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796710 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 19:10 |
| openstack | Launchpad bug 1796710 in tripleo "Tempest tests failed with Read timed out error on tripleo-ci-centos-7-undercloud-containers" [Critical,Triaged] | 19:10 |
| *** slagle has quit IRC | 19:40 | |
| *** agurenko has quit IRC | 19:44 | |
| *** salmankhan has joined #tripleo | 19:48 | |
| *** aedc has quit IRC | 19:48 | |
| *** dciabrin has quit IRC | 19:50 | |
| rook | therve: hey | 19:50 |
| rook | re: mapping to the end of a scale up -- yup. it dropped to 2GB and stayed there after the scale up | 19:51 |
| rook | therve: while it would be nice to understand why - i think the bigger issue is nova-scheduler. | 19:52 |
| mwhahaha | rook: i replied | 20:08 |
| rook | mwhahaha: ack - so that could explain the massive increase | 20:09 |
| mwhahaha | yea | 20:09 |
| rook | dansmith: see mwhahaha's reply to the thread. | 20:09 |
| mwhahaha | instead of 1 you get 5 :D (in my case) | 20:09 |
| rook | to verify, i could simply change that out. | 20:09 |
| rook | yeah, mwhahaha if it is #core i might have 48 | 20:09 |
| rook | lol | 20:09 |
| mwhahaha | yea | 20:09 |
| dansmith | which thread? | 20:09 |
| mwhahaha | fabio was complaining about that recently | 20:09 |
| rook | rhos-dev | 20:09 |
| dansmith | so, yeah the ncpus thing was one thing I was going to bring up | 20:10 |
| dansmith | um, this is an upstream channel, no? :) | 20:10 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 20:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796710 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| mwhahaha | ncpus is always a terrible default | 20:10 |
| openstack | Launchpad bug 1796710 in tripleo "Tempest tests failed with Read timed out error on tripleo-ci-centos-7-undercloud-containers" [Critical,Triaged] | 20:10 |
| *** jtomasek has quit IRC | 20:13 | |
| weshay | mwhahaha, have some odd gate failures | 20:14 |
| weshay | https://bugs.launchpad.net/tripleo/+bug/1796756 | 20:15 |
| mwhahaha | weshay: figures | 20:15 |
| openstack | Launchpad bug 1796756 in tripleo "Error searching for image docker.io/tripleorocky/centos-binary-ceilometer-compute - UnixHTTPConnectionPool(host=\'localhost\', port=None): Read timed out." [Critical,Triaged] | 20:15 |
| weshay | and http://logs.openstack.org/07/608307/1/gate/tripleo-ci-centos-7-containers-multinode/d63b608/job-output.txt.gz#_2018-10-08_18_35_37_437466 | 20:15 |
| weshay | two diff things | 20:15 |
| weshay | gate will reset a few times | 20:15 |
| weshay | also https://bugs.launchpad.net/tripleo/+bug/1796745 | 20:15 |
| openstack | Launchpad bug 1796745 in tripleo "file lock failure on tripleo_common.tests.utils.test_config.TestConfig.test_overcloud_config_one_config_type_StringException" [Critical,Triaged] | 20:15 |
| mwhahaha | first one looks like network issues | 20:16 |
| mwhahaha | http://logs.openstack.org/07/608307/1/gate/tripleo-ci-centos-7-containers-multinode/d63b608/job-output.txt.gz#_2018-10-08_19_40_51_655095 is weird | 20:16 |
| mwhahaha | but seems infra related | 20:16 |
| mwhahaha | bug 1796745 is bad unit testing, i was going to try and fix that mock | 20:16 |
| * mwhahaha blames slagle | 20:16 | |
| weshay | no no.. slagle correctly pointed out that CI was broken | 20:17 |
| mwhahaha | yea because it shouldn't actually be running git in the unit tests | 20:17 |
| *** slagle has joined #tripleo | 20:20 | |
| *** toure is now known as toure|biab | 20:23 | |
| *** slagle has quit IRC | 20:37 | |
| *** ansmith has quit IRC | 20:47 | |
| *** pcaruana has quit IRC | 20:49 | |
| *** raildo has quit IRC | 20:57 | |
| *** trown is now known as trown|outtypewww | 20:59 | |
| *** sai_p has joined #tripleo | 21:04 | |
| *** itlinux has quit IRC | 21:08 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 21:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796710 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796756 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1796710 in tripleo "Tempest tests failed with Read timed out error on tripleo-ci-centos-7-undercloud-containers" [Critical,Triaged] | 21:10 |
| openstack | Launchpad bug 1796756 in tripleo "Error searching for image docker.io/tripleorocky/centos-binary-ceilometer-compute - UnixHTTPConnectionPool(host=\'localhost\', port=None): Read timed out." [Critical,Triaged] | 21:10 |
| roger2 | I am following the instructions at "https://docs.openstack.org/tripleo-quickstart/latest/getting-started.html". I'm on step "Install the Undercloud" (bash quickstart.sh -R master --no-clone --tags all --nodes config/nodes/1ctlr_1comp.yml -I --teardown none -p quickstart-extras-undercloud.yml $VIRTHOST). It fails on "TASK [undercloud-deploy : Install the undercloud]". I replicated the bash command it ran and got same non-zero exit st | 21:12 |
| *** salmankhan has quit IRC | 21:13 | |
| roger2 | The bash command was "sudo /usr/bin/tripleo-container-image-prepare --roles-file /tmp/ansible.xg7NY7-role-data --environment-file /tmp/ansible.tOWpea-prepare-param --cleanup partial 2> ../install-undercloud-manual.log". The log shows something timed-out, "requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)" | 21:13 |
| roger2 | I'm not sure how to troubleshoot further or what I should do. I didn't see anything reported in launchpad that seemed to fit. | 21:14 |
| mwhahaha | we seem to be getting that in ci as well | 21:14 |
| mwhahaha | not sure if it's related to your env or if there is a code problem | 21:14 |
| roger2 | I wonder if I should start over with a named release instead of Master. | 21:17 |
| mwhahaha | yes start with rocky | 21:17 |
| mwhahaha | unless you know what you are doing | 21:17 |
| roger2 | mwhahaha: thank you, I will try rocky. I very don't know what I'm doing. | 21:17 |
| weshay | roger2, you should talk to EmilienM then :) | 21:18 |
| * roger2 waves at EmilienM | 21:18 | |
| weshay | roger2, I kid a little, anyway we try to have a little fun | 21:18 |
| roger2 | weshay: sounds good to me | 21:19 |
| weshay | roger2, so the issue you are hitting is when the undercloud is downloading containers, and updating them w/ the latest rpms etc.. | 21:19 |
| weshay | roger2, def should not use master, rocky is better, queens might be your best bet | 21:19 |
| mwhahaha | i said rocky cause the upstream docs are more likely to work with rocky than queens | 21:20 |
| roger2 | weshay: good info, thank you | 21:20 |
| mwhahaha | as we don't branch our docs | 21:20 |
| roger2 | mwhahaha: so thats why docs didn't specify a release | 21:20 |
| roger2 | well, aside from Master | 21:21 |
| mwhahaha | yea we usually do stable branch designations | 21:21 |
| mwhahaha | but i've noticed we've not kept up on the differences of the container commands and stuff | 21:21 |
| weshay | too bad the ci doesn't doc it | 21:22 |
| * weshay runs | 21:22 | |
| mwhahaha | weshay: https://review.openstack.org/#/c/608774/ | 21:22 |
| mwhahaha | i fixed it for you | 21:22 |
| weshay | thanks.. aint for me, but thanks | 21:23 |
| mwhahaha | it's always for you | 21:23 |
| mwhahaha | <3 | 21:23 |
| weshay | lolz | 21:23 |
| roger2 | I wish I could deploy rocky with containerized UC and OC on my baremetal nodes with ironic introspection and automatic role assignment, but documentation is too hard for me at this point in my learning. So I'm trying to just do what the doc says for now to gain experience. | 21:24 |
| mwhahaha | containerized UC should be pretty straight forward for rocky | 21:25 |
| mwhahaha | the ironic bits could get messy | 21:25 |
| mwhahaha | roger2: https://docs.openstack.org/tripleo-docs/latest/install/installation/installation.html are the directions for containerized undercloud (if >= Rocky) | 21:27 |
| weshay | mwhahaha, openstack/tripleo-common-tempest-plugin is that for ui tests? | 21:27 |
| mwhahaha | weshay: no it was for the tripleo-common "api" stuff | 21:28 |
| mwhahaha | weshay: the ui one was the thing honza was doing | 21:28 |
| mwhahaha | tripleo-ui-tempest-plugin | 21:28 |
| weshay | mwhahaha, can we -2 patches until we see integration tests? | 21:28 |
| mwhahaha | integration tests for what? | 21:28 |
| mwhahaha | tripleo-common-tempest-plugin is tempest tests but for tripleo-common | 21:29 |
| mwhahaha | it's not currently wired up ( don't think there are any tests actually) | 21:29 |
| weshay | ya.. roger that | 21:29 |
| weshay | ok.. at some point.. I don't know when that point is | 21:29 |
| weshay | however I don't want to repeat tripleo-upgrades | 21:29 |
| weshay | one test.. even if it fails is better than just tox tests.. at some point.. | 21:30 |
| weshay | anyhoo | 21:30 |
| mwhahaha | roger2: https://docs.openstack.org/tripleo-docs/latest/install/basic_deployment/basic_deployment_cli.html should be accurate up until the deploy overcloud bits. but you need to do https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/overcloud.html before you do the overcloud deploy | 21:30 |
| mwhahaha | weshay: i don't know who is driving tripleo-common-tempest-plugin | 21:31 |
| roger2 | mwhahaha: ah, thank you. so it sounds like quickstart.sh isn't really what I need. | 21:32 |
| mwhahaha | roger2: if you have baremetal hosts, no it is not really. it would be better to just follow the instructions | 21:32 |
| mwhahaha | roger2: quickstart is good if you want an environment up to test things | 21:33 |
| roger2 | mwhahaha: thank you for the good advice. | 21:34 |
| mwhahaha | weshay: so that UnixHTTPConntionPool thing is weird because the journal log just stops like 10 mins prior to the ansible failing, http://logs.openstack.org/68/607368/1/gate/tripleo-ci-centos-7-containers-multinode/e2d32d5/logs/undercloud/var/log/journal.txt.gz#_Oct_08_18_47_05 | 21:37 |
| mwhahaha | weshay: do we capture a df in the log collection | 21:39 |
| mwhahaha | i'm wondering if we ran out of space | 21:39 |
| weshay | heh | 21:41 |
| *** agopi has quit IRC | 21:42 | |
| weshay | what the | 21:43 |
| weshay | all kinds of whacky errors | 21:43 |
| weshay | 47:29.851 ERROR /var/log/containers/neutron/dhcp-agent.log: 57050 ERROR neutron.agent.dhcp.agent /usr/bin/docker-current: Error response from daemon: Conflict. The container name "/neutron-dnsmasq-qdhcp-0f8c81ae-46f3-4cbb-b4b2-ffc5e01d674b" is already in use by container ef19279d0ccaee53ad041c7387dccb730cb17fdcb96bd20d5371770a98cd802f. You have to remove (or rename) that container to be able to reuse that name.. | 21:43 |
| weshay | 2018-10-08 19:03:38.051 ERROR /var/log/containers/keystone/keystone.log: 393 ERROR keystone.assignment.core [req-ba56a8ff-da48-400f-a1ac-6afe472ede3c - - - - -] Circular reference found role inference rules - 7e10a845a7894393b9f69d6344ebe92b. | 21:43 |
| weshay | 018-10-08 19:00:02.520 ERROR /var/log/containers/neutron/openvswitch-agent.log: 57685 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.br_int [req-ac795b1c-2e33-41f9-8600-1c6cec72d11b - - - - -] Failed to communicate with the switch: RuntimeError: ofctl request version=None,msg_type=None,msg_len=None,xid=None,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294 | 21:43 |
| weshay | 967295,table_id=23,type=1) error Datapath Invalid 156397689906499 | 21:43 |
| weshay | another instances of image-prepare failing | 21:46 |
| weshay | http://logs.openstack.org/64/604664/1/gate/tripleo-ci-centos-7-containers-multinode/3f8c81e/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-10-08_20_46_53 | 21:46 |
| weshay | OperationalError: (sqlite3.OperationalError) database is locked [SQL: "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"] (Background on this error at: http://sqlalche.me/e/e3q8) | 21:47 |
| * weshay wonders if a provider is going down | 21:47 | |
| mwhahaha | not completely sure | 21:48 |
| *** rh-jelabarre has quit IRC | 21:48 | |
| weshay | no trends yet.. but the data is not updated http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=61&fullscreen | 21:48 |
| mwhahaha | i would wonder if it was related to our increased process count for it, but it was perfectly fine for over a week. this seems new | 21:49 |
| weshay | it's too bad we don't have +2 on elastic search queries | 21:51 |
| weshay | this would be SOO much easier | 21:51 |
| weshay | going to ask clark and doug for that on wed.. mwhahaha have a meeting w/ infra folks | 21:52 |
| weshay | about mentorship | 21:52 |
| weshay | mentoring / mentorship .. meh | 21:52 |
| weshay | https://docs.sqlalchemy.org/en/latest/errors.html#error-e3q8 | 21:53 |
| weshay | is that ara crapping out on us? | 21:54 |
| mwhahaha | might be unrealted | 21:55 |
| weshay | File "/usr/lib/python2.7/site-packages/ara/plugins/callbacks/log_ara.py", line 41, in <module> | 21:55 |
| *** aedc has joined #tripleo | 21:56 | |
| mwhahaha | weshay: interesting, ara shouldn't interfere but i can totally see it not working with this because we run multiple playbook executions at once | 21:59 |
| weshay | mwhahaha, has that always been the case? | 22:00 |
| mwhahaha | weshay: so yea i think ara failed on that one | 22:00 |
| weshay | k.. I had a patch to turn ara off | 22:00 |
| weshay | for the overcloud | 22:00 |
| mwhahaha | weshay: we didn't run ara until recently, and when we increased the process count that might affect it | 22:00 |
| mwhahaha | concurrency problems | 22:00 |
| *** slaweq has quit IRC | 22:01 | |
| weshay | I blame openstack, ara is an official project now | 22:02 |
| *** agopi has joined #tripleo | 22:05 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796710 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796756 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796764 | 22:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1796710 in tripleo "Tempest tests failed with Read timed out error on tripleo-ci-centos-7-undercloud-containers" [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1796756 in tripleo "Error searching for image docker.io/tripleorocky/centos-binary-ceilometer-compute - UnixHTTPConnectionPool(host=\'localhost\', port=None): Read timed out." [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1796764 in tripleo "OperationalError: (sqlite3.OperationalError) database is locked [SQL: "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"] (Background on this error at: http://sqlalche.me/e/e3q8)" [Critical,Triaged] | 22:10 |
| *** boazel_ has joined #tripleo | 22:11 | |
| *** devep has joined #tripleo | 22:12 | |
| *** devep has quit IRC | 22:12 | |
| *** boazel has quit IRC | 22:14 | |
| honza | weshay: anything i can help clear up? | 22:16 |
| weshay | honza, as we bring up new repos under tripleo we just need to ensure there is sufficient tests | 22:19 |
| * weshay is just double checking some of the latest new repos | 22:19 | |
| weshay | the tempest plugin for tripleo-common is new info for me | 22:20 |
| weshay | honza, I think I'm fairly familiar w/ the ui work, tempest-plugin for ui using selenium etc | 22:20 |
| weshay | gate reset | 22:21 |
| weshay | gate reset | 22:22 |
| weshay | dang it.. zuul ui is down | 22:22 |
| *** boazel_ has quit IRC | 22:24 | |
| weshay | honza, for instance.. can you put in a change that triggers https://review.openstack.org/#/c/607327/ | 22:24 |
| *** akhila has joined #tripleo | 22:25 | |
| weshay | curl http://zuul.openstack.org/status works | 22:28 |
| *** ansmith has joined #tripleo | 22:35 | |
| *** rcernin has joined #tripleo | 22:50 | |
| *** tosky has quit IRC | 23:03 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 23:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796710 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796756 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796764 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1796710 in tripleo "Tempest tests failed with Read timed out error on tripleo-ci-centos-7-undercloud-containers" [Critical,Triaged] | 23:10 |
| openstack | Launchpad bug 1796756 in tripleo "Error searching for image docker.io/tripleorocky/centos-binary-ceilometer-compute - UnixHTTPConnectionPool(host=\'localhost\', port=None): Read timed out." [Critical,Triaged] | 23:10 |
| openstack | Launchpad bug 1796764 in tripleo "OperationalError: (sqlite3.OperationalError) database is locked [SQL: "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"] (Background on this error at: http://sqlalche.me/e/e3q8)" [Critical,Triaged] | 23:10 |
| *** rlandy is now known as rlandy|bbl | 23:11 | |
| *** quiquell|off has quit IRC | 23:27 | |
| *** akhila has quit IRC | 23:31 | |
| *** artom has joined #tripleo | 23:34 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
