| *** bfournie has quit IRC | 00:00 | |
| *** bfournie has joined #tripleo | 00:01 | |
| *** moshele has quit IRC | 00:02 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: ADD MTU settings and Neutron settings adjustment https://review.openstack.org/527249 | 00:12 |
|---|---|---|
| *** threestrands has joined #tripleo | 00:26 | |
| *** threestrands has quit IRC | 00:26 | |
| *** threestrands has joined #tripleo | 00:26 | |
| *** slacko_16322 has joined #tripleo | 00:58 | |
| *** itlinux has joined #tripleo | 00:59 | |
| openstackgerrit | zenghui.shi proposed openstack/tripleo-heat-templates master: Add PTP composable service https://review.openstack.org/491317 | 01:01 |
| *** dhill_ has quit IRC | 01:11 | |
| mwhahaha | dmsimard: more dns problems, http://logs.openstack.org/39/526439/5/gate/tripleo-ci-centos-7-containers-multinode/f55c04f/logs/undercloud/home/zuul/vxlan_networking.sh.log.txt.gz#_2017-12-19_23_57_37 | 01:11 |
| dmsimard | mwhahaha: is that occurring inside a container ? | 01:12 |
| mwhahaha | dmsimard: no | 01:13 |
| mwhahaha | Is undercloud setup bits | 01:13 |
| mwhahaha | Our vxlan config or whatever | 01:13 |
| dmsimard | Ok, keep sending those, I have a list | 01:14 |
| *** chem has quit IRC | 01:15 | |
| *** psachin has joined #tripleo | 01:24 | |
| *** cshastri has joined #tripleo | 01:26 | |
| *** psachin has quit IRC | 01:28 | |
| *** psachin has joined #tripleo | 01:33 | |
| *** slacko_16322 has quit IRC | 01:37 | |
| *** yamahata has quit IRC | 01:38 | |
| *** yamahata has joined #tripleo | 01:39 | |
| *** gfidente|afk has quit IRC | 01:39 | |
| *** jd_ has quit IRC | 01:45 | |
| *** jd_ has joined #tripleo | 01:47 | |
| *** dmacpher has joined #tripleo | 01:47 | |
| *** agopi has joined #tripleo | 01:52 | |
| *** jongwooh has joined #tripleo | 02:01 | |
| *** dprince has quit IRC | 02:06 | |
| *** karthiks has joined #tripleo | 02:16 | |
| itlinux | hello all.. I wonder about this issues.. Introspection of node 13a27ec6-5167-456e-a081-dfd076f48639 timed out. | 02:23 |
| itlinux | I had ocata running and now trying to run pike I upgraded the UC fine.. | 02:23 |
| itlinux | any tips on this.. since the other look c04c86f6-3024-4590-b294-512cecfcf53d | None | None | power off | available | Tru | 02:24 |
| itlinux | thanks | 02:24 |
| *** jlabarre has quit IRC | 02:26 | |
| *** fzdarsky_ has joined #tripleo | 02:29 | |
| jongwooh | how is the result of "openstack baremetal instrospection status <uuid>"? | 02:29 |
| *** fzdarsky has quit IRC | 02:30 | |
| *** Goneri has quit IRC | 02:30 | |
| *** catintheroof has joined #tripleo | 02:38 | |
| *** atarlov has joined #tripleo | 02:48 | |
| itlinux | let me check | 02:49 |
| itlinux | http://paste.openstack.org/show/629417/ | 02:50 |
| itlinux | running the retry now.. | 02:51 |
| itlinux | Introspection of node 13a27ec6-5167-456e-a081-dfd076f48639 timed out. | 02:51 |
| *** catintheroof has quit IRC | 02:53 | |
| *** rlandy|rover has quit IRC | 03:01 | |
| *** threestrands has quit IRC | 03:03 | |
| *** threestrands has joined #tripleo | 03:04 | |
| itlinux | it comes back like this 13a27ec6-5167-456e-a081-dfd076f48639 | None | None | None | enroll | False | 03:04 |
| *** threestrands has quit IRC | 03:05 | |
| *** threestrands has joined #tripleo | 03:06 | |
| *** threestrands has joined #tripleo | 03:06 | |
| *** threestrands has quit IRC | 03:07 | |
| *** threestrands has joined #tripleo | 03:07 | |
| itlinux | looks like | last_error | Failed to change power state to 'power on' by 'rebooting'. Error: IPMI | | 03:08 |
| itlinux | | | call failed: power status. | 03:08 |
| itlinux | jongwooh: any tips on that.. | 03:09 |
| *** yamahata has quit IRC | 03:11 | |
| *** karthiks has quit IRC | 03:23 | |
| *** psahoo has joined #tripleo | 03:24 | |
| itlinux | I think I found the issue I cannot ipmi to the box! | 03:25 |
| *** artom has quit IRC | 03:31 | |
| *** artom has joined #tripleo | 03:31 | |
| jongwooh | ok you found it | 03:40 |
| *** ramishra has joined #tripleo | 03:50 | |
| *** owalsh_ has joined #tripleo | 03:55 | |
| *** udesale has joined #tripleo | 03:57 | |
| *** threestrands_ has joined #tripleo | 03:57 | |
| *** threestrands has quit IRC | 03:57 | |
| *** threestrands_ has quit IRC | 03:58 | |
| *** threestrands_ has joined #tripleo | 03:59 | |
| *** owalsh has quit IRC | 03:59 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Use skopeo for tag discover https://review.openstack.org/528945 | 04:06 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Move more prepare logic into kolla_builder https://review.openstack.org/526579 | 04:06 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Use push_destination as the registry host in env file https://review.openstack.org/528616 | 04:06 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Prepare action: extra arguments https://review.openstack.org/526580 | 04:06 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: WIP Discover every tag on prepare https://review.openstack.org/529215 | 04:06 |
| *** liverpooler has quit IRC | 04:14 | |
| *** shreshtha has joined #tripleo | 04:15 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Add the option to run the container-check script https://review.openstack.org/501028 | 04:21 |
| *** ykarel has joined #tripleo | 04:27 | |
| *** pgadiya has joined #tripleo | 04:29 | |
| *** pgadiya has quit IRC | 04:39 | |
| *** pdeore has joined #tripleo | 04:42 | |
| *** gvrangan_ has joined #tripleo | 04:46 | |
| *** dpawar has joined #tripleo | 04:47 | |
| *** gvrangan_ has quit IRC | 04:55 | |
| *** pgadiya has joined #tripleo | 04:56 | |
| *** ianw is now known as ianw_pto | 04:57 | |
| *** links has joined #tripleo | 04:57 | |
| openstackgerrit | Merged openstack/tripleo-common master: Inital import of tripleo ansible inventory code https://review.openstack.org/528342 | 05:03 |
| *** moshele has joined #tripleo | 05:15 | |
| *** skramaja has joined #tripleo | 05:27 | |
| *** pgadiya has quit IRC | 05:30 | |
| *** pgadiya has joined #tripleo | 05:31 | |
| *** gkadam has joined #tripleo | 05:43 | |
| openstackgerrit | Merged openstack/instack-undercloud master: Add missing include of ironic::drivers::ansible https://review.openstack.org/526439 | 05:51 |
| Tengu | hello there :) | 06:05 |
| Tengu | EmilienM: if you're still up: time to sleep ;) | 06:05 |
| *** rbrady has quit IRC | 06:05 | |
| Tengu | mwhahaha: if you're still here, can you point me some location for documenting the new basic auth feature in haproxy? | 06:05 |
| *** rbrady has joined #tripleo | 06:06 | |
| *** rbrady has joined #tripleo | 06:06 | |
| *** psahoo has quit IRC | 06:11 | |
| *** marios has joined #tripleo | 06:16 | |
| *** psahoo has joined #tripleo | 06:16 | |
| *** d0ugal has quit IRC | 06:17 | |
| *** jaganathan has joined #tripleo | 06:19 | |
| *** d0ugal has joined #tripleo | 06:22 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Fix up the rabbitmq-ready check https://review.openstack.org/527403 | 06:24 |
| *** jfrancoa has joined #tripleo | 06:38 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-quickstart master: Do not use puppet-ceph on newton https://review.openstack.org/529234 | 06:41 |
| *** agopi has quit IRC | 06:42 | |
| *** agopi has joined #tripleo | 06:42 | |
| *** masco has joined #tripleo | 06:43 | |
| *** karthiks has joined #tripleo | 06:49 | |
| *** janki has joined #tripleo | 06:49 | |
| *** karthiks has quit IRC | 06:54 | |
| *** threestrands_ has quit IRC | 06:57 | |
| *** pdeore has quit IRC | 06:58 | |
| *** agurenko has joined #tripleo | 06:59 | |
| *** pdeore has joined #tripleo | 07:05 | |
| *** karthiks has joined #tripleo | 07:06 | |
| *** rcernin has quit IRC | 07:08 | |
| *** gkadam has quit IRC | 07:09 | |
| *** dsneddon has quit IRC | 07:12 | |
| *** yprokule has joined #tripleo | 07:12 | |
| *** cylopez has joined #tripleo | 07:22 | |
| *** holser__ has joined #tripleo | 07:28 | |
| *** shardy has joined #tripleo | 07:29 | |
| *** dmacpher has quit IRC | 07:34 | |
| *** agopi has quit IRC | 07:38 | |
| *** agopi has joined #tripleo | 07:38 | |
| *** abregman has joined #tripleo | 07:42 | |
| *** ebarrera has joined #tripleo | 08:00 | |
| *** stendulker has joined #tripleo | 08:06 | |
| *** nyechiel has joined #tripleo | 08:09 | |
| *** psahoo has quit IRC | 08:14 | |
| moshele | janki: hi | 08:23 |
| janki | moshele, hey | 08:24 |
| moshele | janki: can we talk in bluejeans I have some questions issues with opendaylight deployment | 08:25 |
| janki | moshele, I have few things lined up. HOw about in an hour? | 08:26 |
| *** cshastri has quit IRC | 08:26 | |
| moshele | janki: sure ping me when you can | 08:27 |
| *** jtomasek has joined #tripleo | 08:27 | |
| *** ccamacho has joined #tripleo | 08:27 | |
| janki | moshele, sure and about yesterday's query, there is a dependent ODL patch that is needed - https://git.opendaylight.org/gerrit/#/c/64602/ | 08:29 |
| *** psahoo has joined #tripleo | 08:30 | |
| *** jtomasek has quit IRC | 08:31 | |
| *** jtomasek has joined #tripleo | 08:32 | |
| *** gkadam has joined #tripleo | 08:32 | |
| *** amoralej|off is now known as amoralej | 08:35 | |
| sri_ | mwhahaha, got it thanks | 08:37 |
| Tengu | hello! | 08:38 |
| Tengu | small question: is this doc still up-to-date? https://docs.openstack.org/tripleo-docs/latest/install/post_deployment/quiesce_compute.html#quiesce-compute | 08:38 |
| *** cshastri has joined #tripleo | 08:39 | |
| Tengu | apparently, nova account has a ~/.ssh/config that points to a command wrapper and a distinct SSH port, enforcing "nova_migration" user. | 08:39 |
| Tengu | meaning: the ssh key won't be used. | 08:39 |
| *** agurenko has quit IRC | 08:40 | |
| *** jpena|off is now known as jpena | 08:44 | |
| *** paramite has joined #tripleo | 08:47 | |
| *** pgadiya has quit IRC | 08:50 | |
| *** mdnadeem has joined #tripleo | 08:51 | |
| *** ukalifon has joined #tripleo | 08:52 | |
| *** psahoo has quit IRC | 08:56 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DNM: test container updates https://review.openstack.org/515372 | 08:57 |
| *** anilvenkata has joined #tripleo | 08:58 | |
| *** hjensas has quit IRC | 09:01 | |
| *** chem has joined #tripleo | 09:03 | |
| *** pgadiya has joined #tripleo | 09:04 | |
| skramaja | shardy: hi, are you working on modifying the deprecated params workflow according the heat based env merging? | 09:04 |
| skramaja | shardy: i am planning to add a validation for role-specific parameters in the same workflow, if you in progress, i will wait for it to complete. | 09:04 |
| shardy | skramaja: Hi, no I haven't got to that yet - to be honest the work to move environment merging to heat stalled when I started modifying tripleo-common, because I ran into some request limit problems with heat | 09:05 |
| shardy | skramaja: I'd like to get back to it, but it probably requires changes to heat to load files directly from swift | 09:05 |
| shardy | skramaja: so please feel free to go ahead and make your workflow changes :) | 09:06 |
| skramaja | sure shardy | 09:07 |
| shardy | Tengu: probably owalsh_ is your best contact for the nova migration questions | 09:08 |
| *** psahoo has joined #tripleo | 09:09 | |
| Tengu | shardy: hmm ok. well, I could "fake" it using openstack server migrate --wait --block-migration --live <dest> <id>. As we "only" have 2 computes, evacuating one isn't hard. | 09:10 |
| owalsh_ | Tengu: nope, docs are not up to date - https://review.openstack.org/499543 | 09:13 |
| Tengu | owalsh_: ah, thanks :) | 09:14 |
| shardy | Thanks owalsh_:) | 09:14 |
| Tengu | owalsh_: would be great to release that change :) | 09:15 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Configure qemu group setting as hugetlbfs for ovs-dpdk https://review.openstack.org/529272 | 09:15 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Removed ovs-dpdk workaround to fix the vhost socket permission https://review.openstack.org/529273 | 09:15 |
| owalsh_ | Tengu: indeed :-) I'll take a look at it today | 09:17 |
| *** owalsh_ is now known as owalsh | 09:17 | |
| Tengu | owalsh: thank you :). In the meantime, I'm migrating nodes one by one with the `server migrate' command. | 09:18 |
| Tengu | wokring well so far. | 09:18 |
| openstackgerrit | Juan Badia Payno proposed openstack/tripleo-heat-templates master: logging: use service_config_settings for fluentd https://review.openstack.org/501458 | 09:23 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Update templates alias to queens https://review.openstack.org/529276 | 09:27 |
| *** agopi has quit IRC | 09:27 | |
| *** oidgar has joined #tripleo | 09:28 | |
| *** anilvenkata has quit IRC | 09:28 | |
| *** agurenko has joined #tripleo | 09:28 | |
| oidgar | hi everybody, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master and gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master fails for me for a long time, but I'm not sure if this related to RDO cloud issues from the last days | 09:29 |
| *** anilvenkata has joined #tripleo | 09:29 | |
| oidgar | It fails because it cannot find brctl command | 09:29 |
| *** dsariel has joined #tripleo | 09:29 | |
| oidgar | anyone here has experience with those gates? | 09:29 |
| *** fragatina has joined #tripleo | 09:30 | |
| *** fragatina has quit IRC | 09:31 | |
| *** fragatina has joined #tripleo | 09:31 | |
| *** lucas-afk is now known as lucasagomes | 09:34 | |
| *** aditya_r has joined #tripleo | 09:34 | |
| honza | mandre: i tried to install an undercloud the old way in hopes of getting a proper hiera instance, but then i run into IP address change issues when using run.sh | 09:44 |
| *** derekh has joined #tripleo | 09:44 | |
| mandre | honza: you need to tweak your undercloud.conf i believe | 09:45 |
| *** aditya_ra has joined #tripleo | 09:45 | |
| honza | mandre: did you see my messages from last night about the controller_admin_host/hiera issues? | 09:45 |
| mandre | honza: I didn't | 09:46 |
| honza | mandre: I'm getting a bunch of hiera-related issues, and I remember you telling me about a hack I needed. I searched my irc logs and emails but couldn't find it. | 09:46 |
| honza | mandre: http://paste.openstack.org/show/629388/ | 09:47 |
| mandre | honza: oh... that! you need a newer puppet-tripleo module | 09:48 |
| honza | !!! | 09:48 |
| openstack | honza: Error: "!!" is not a valid command. | 09:48 |
| honza | openstack: lol | 09:48 |
| *** aditya_r has quit IRC | 09:48 | |
| honza | mandre: how can i get a newer one? | 09:49 |
| honza | it's commented out! | 09:49 |
| honza | oh my | 09:49 |
| mandre | honza: https://review.openstack.org/#/c/525761/ | 09:49 |
| honza | mandre: thank you so much | 09:50 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-docs master: Remove obsolete section on compute ssh-key setup https://review.openstack.org/499543 | 09:51 |
| mandre | honza: just uncomment https://github.com/dprince/undercloud_containers/blob/master/doit.sh#L145-L153 and that should get you puppet-tripleo from a checkout | 09:52 |
| moshele | janki: I hope you haven't forgot me ; ) | 09:57 |
| *** cylopez has left #tripleo | 09:58 | |
| *** florianf has joined #tripleo | 09:58 | |
| janki | moshele, ofcourse not. give me 10 more minutes plz | 10:00 |
| moshele | janki: sure | 10:00 |
| lyarwood | alee: pingo, re https://review.openstack.org/#/c/526514/ did you also have a THT change enabling this? | 10:00 |
| *** psachin has quit IRC | 10:01 | |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Add docker-registry service https://review.openstack.org/526132 | 10:03 |
| *** tosky has joined #tripleo | 10:05 | |
| *** nyechiel has quit IRC | 10:08 | |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: nova: Add VerifyGlanceSignatures compute param https://review.openstack.org/529286 | 10:11 |
| *** psachin has joined #tripleo | 10:11 | |
| lyarwood | alee: ^ https://review.openstack.org/529286 - nova-compute THT change to introduce a param for this, let me know if you already have something and I'll kill this change. | 10:11 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-docs master: Remove obsolete section on compute ssh-key setup https://review.openstack.org/499543 | 10:11 |
| owalsh | lyarwood: looks like alee has a related review https://review.openstack.org/527136 | 10:14 |
| *** jeblair has quit IRC | 10:15 | |
| *** bcafarel has quit IRC | 10:15 | |
| owalsh | lyarwood: NB the potential migration issues | 10:15 |
| *** jeblair has joined #tripleo | 10:16 | |
| Tengu | owalsh: small question: can we migrate a stopped instance? | 10:16 |
| * lyarwood waits for gerrit's webui to load | 10:17 | |
| owalsh | Tengu: cold migration should work AFAIK | 10:19 |
| lyarwood | owalsh: ah, using ExtraConfig, would be nice to have a param in THT tbh | 10:19 |
| Tengu | owalsh: hmm ok. had some issues with that one, and apparently "live" is working fine. will stop services and launch the live migration. | 10:20 |
| *** psachin has quit IRC | 10:20 | |
| owalsh | Tengu: stop services? Need everything running for live migration to work I would think | 10:23 |
| Tengu | owalsh: ah, services in the instance, not node | 10:23 |
| *** shardy has quit IRC | 10:24 | |
| owalsh | Tengu: ah, ack. Probably will work without stopping services on the instances but certain services don't like live-migration (e.g rabbitmq) | 10:24 |
| Tengu | owalsh: or docker/rancher :) | 10:24 |
| Tengu | just stopped all docker containers, migration's running, and that's it. we don't have prod per se on the openstack, I can do some small downtime :). | 10:25 |
| *** salmankhan has joined #tripleo | 10:26 | |
| owalsh | Tengu: yea, cold migration might be more appropriate if there is something like rancher running on top. Do you recall what the issue was with cold migration? I landed a few fixes a while back | 10:27 |
| Tengu | owalsh: an issue with the root-wrap thingy, unfortunately I can't get the command output because its log is squashed - that lead me to create this review: https://review.openstack.org/#/c/518695/ but apparently, it doesn't suit people from Oslo :( | 10:28 |
| Tengu | I think I also opened an issue one LP, wait. | 10:29 |
| *** dciabrin__ has joined #tripleo | 10:30 | |
| *** dciabrin_ has quit IRC | 10:30 | |
| Tengu | ah, related issue, owalsh : https://bugs.launchpad.net/oslo.concurrency/+bug/1731185 | 10:31 |
| openstack | Launchpad bug 1731185 in oslo.concurrency "Not enough debug info for "execute"" [Undecided,In progress] - Assigned to Cédric Jeanneret (cjeanneret-c2c) | 10:31 |
| Tengu | but I was more searching for debug logs. | 10:31 |
| owalsh | Tengu: can't seem to login to rdo gerrit but the patches were https://github.com/rdo-packages/nova-distgit/commit/c34374a867cf022e8c8773ab46ac1a032aa9d29e & https://github.com/rdo-packages/nova-distgit/commit/2955f70ce42e4f62ed7661817ed0c2a1dce600e8 | 10:34 |
| *** hewbrocca_afk is now known as hewbrocca | 10:34 | |
| Tengu | owalsh: I'll check that - I can't update our current openstack deploy due to the lack of proper lab for update testing, but that's planned. | 10:35 |
| Tengu | owalsh: but your patches seem to meet the thing I stumbled upon, nice catch! | 10:35 |
| Tengu | owalsh: was it backported in Pike? | 10:36 |
| owalsh | Tengu: yes, think it was backported to Pike and Newton. Looking at the LP though I don't think that's the issue as it is failing in the touch command | 10:36 |
| Tengu | owalsh: hmm. you're right. it was "some time" ago, I don't recall all the details unfortunately. | 10:37 |
| Tengu | and as we're in the (urgent) need to move instance around, I can't afford to re-create this issue right now. | 10:37 |
| Tengu | I have to free one of our two computes in order to reinstall it properly. | 10:37 |
| owalsh | Tengu: np, just interested in (or to blame for) any issues with this | 10:38 |
| Tengu | owalsh: *taking notes* :) | 10:38 |
| Tengu | owalsh: next year I'll be able to test that in better conditions, as we'll integrate 2 new nodes (meaning 4 computes), hence more way to play with instances around. | 10:39 |
| *** dciabrin__ has quit IRC | 10:39 | |
| Tengu | owalsh: so I might ping you back then | 10:39 |
| owalsh | Tengu: sure | 10:39 |
| owalsh | Tengu: just FYI while it's fresh in my head... check /var/lib/nova/.ssh/config looks like https://github.com/rdo-packages/nova-distgit/blob/rpm-master/nova-ssh-config | 10:40 |
| Tengu | owalsh: 2s, I think it's the same content, just need to ensure that | 10:40 |
| owalsh | Tengu: and check the target IP address is allowed in the Match block in /etc/ssh/sshd_config | 10:41 |
| *** dciabrin has joined #tripleo | 10:41 | |
| owalsh | Tengu: I expect it's one of those files if the touch command is failing | 10:41 |
| Tengu | ah, nope. a bit more lines in it. maybe I added them: http://paste.openstack.org/show/629445/ also, port… ?! | 10:41 |
| Tengu | owalsh: the match blocs: http://paste.openstack.org/show/629446/ | 10:42 |
| *** bcafarel has joined #tripleo | 10:42 | |
| owalsh | Tengu: port 2022 is for containers only IIRC | 10:42 |
| Tengu | o_O errr… we didn't deploy with containers… | 10:43 |
| Tengu | ah, but sshd is listening on 22 and 2022 | 10:43 |
| Tengu | so not a problem | 10:43 |
| *** hjensas has joined #tripleo | 10:43 | |
| *** hjensas has quit IRC | 10:43 | |
| *** hjensas has joined #tripleo | 10:43 | |
| Tengu | hmmm | 10:43 |
| Tengu | ah, ok, so if a "nova_migration" user hits ssh, it checks if the request IP is 101.6, else drop. didn't understand it correctly. the blocks are OK I think. | 10:45 |
| owalsh | Tengu: yes | 10:45 |
| owalsh | Tengu: do the keys exist? Maybe ssh isn't setup at all but live migration isn't using it (i.e it's live_migration_uri isn't set to qemu+ssh in nova.conf) | 10:46 |
| * owalsh really needs to write up all of the details somewhere | 10:50 | |
| Tengu | owalsh: I've checked the key existence back then, and yep, they do exist, and are allowed as well | 10:50 |
| Tengu | but without a proper command log output, I just can't check anything. | 10:51 |
| Tengu | owalsh: I suspect the "/sbin/nologin" to be maybe an issue though | 10:51 |
| owalsh | Tengu: should be /bin/bash for the nova_migration user, /sbin/nologin for nova | 10:52 |
| Tengu | ah, yes, true. different user. | 10:52 |
| Tengu | -.- tricky. | 10:52 |
| owalsh | Tengu: I think I'll flesh out https://review.openstack.org/499543 with more details on the setup and how to test it | 10:52 |
| Tengu | owalsh: good idea :). | 10:53 |
| Tengu | owalsh: I took the time to check how things were supposed to work for the migration, but I'm not sure I could get all the things. | 10:54 |
| Tengu | have to go, end-of-year dinner with the colleagues. of course, as I'm in Switzerland, Fondue time :). | 10:54 |
| owalsh | Tengu; enjoy | 10:57 |
| *** cshastri has quit IRC | 10:57 | |
| *** agurenko has quit IRC | 10:58 | |
| *** dtantsur|afk is now known as dtantsur | 11:02 | |
| *** yolanda__ has joined #tripleo | 11:05 | |
| *** hewbrocca is now known as hewbrocca_afk | 11:07 | |
| *** yolanda has quit IRC | 11:08 | |
| *** cshastri has joined #tripleo | 11:10 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: [WIP] Add a report of the Workflow execution on failure https://review.openstack.org/526653 | 11:15 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Create flavors for undercloud https://review.openstack.org/526810 | 11:21 |
| *** moshele has quit IRC | 11:22 | |
| *** stendulker has quit IRC | 11:25 | |
| *** pdeore has quit IRC | 11:26 | |
| *** cshastri has quit IRC | 11:29 | |
| *** nyechiel has joined #tripleo | 11:30 | |
| *** caboucha has joined #tripleo | 11:31 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Ignore errors in graphite task https://review.openstack.org/529302 | 11:31 |
| sshnaidm | cores, please some urgent fix ^^ | 11:31 |
| sshnaidm | trown|outtypewww, panda ^^ | 11:31 |
| *** pdeore has joined #tripleo | 11:36 | |
| *** oidgar has quit IRC | 11:40 | |
| *** psachin has joined #tripleo | 11:40 | |
| *** agurenko has joined #tripleo | 11:45 | |
| *** akrivoka has joined #tripleo | 11:54 | |
| *** aditya_ra has quit IRC | 11:54 | |
| *** aditya_ra has joined #tripleo | 11:54 | |
| *** oidgar has joined #tripleo | 11:56 | |
| oidgar | hi, does anyone else here encounter non stop gate failures in tripleo-common patches? | 11:57 |
| *** dciabrin has quit IRC | 12:00 | |
| *** shreshtha has quit IRC | 12:04 | |
| *** salmankhan has quit IRC | 12:04 | |
| *** salmankhan has joined #tripleo | 12:06 | |
| *** aditya_ra has quit IRC | 12:14 | |
| *** dciabrin has joined #tripleo | 12:18 | |
| d0ugal | oidgar: do you have an example? | 12:23 |
| *** bfournie has quit IRC | 12:23 | |
| *** bfournie has joined #tripleo | 12:23 | |
| *** moshele has joined #tripleo | 12:23 | |
| *** dpawar has quit IRC | 12:26 | |
| *** lucasagomes is now known as lucas-hungry | 12:26 | |
| *** bfournie has quit IRC | 12:28 | |
| oidgar | d0ugal: 1s, upstream gerrit returns "service unavailable..." | 12:28 |
| *** jlabarre has joined #tripleo | 12:29 | |
| oidgar | d0ugal: https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/900/ | 12:29 |
| *** raildo has joined #tripleo | 12:29 | |
| *** pdeore has quit IRC | 12:29 | |
| oidgar | hjensas: do we have a meeting now? | 12:32 |
| d0ugal | oidgar: so it is the 3rd party CI? | 12:32 |
| oidgar | d0ugal: yes, it fails consistently in the last week | 12:33 |
| d0ugal | oidgar: I think you can ignore the third party CI. I believe it is unstable and hopefully still being worked on | 12:33 |
| d0ugal | I don't really think I have ever seen it pass reliably yet | 12:33 |
| d0ugal | but I'm not sure who is responsible for it. | 12:33 |
| oidgar | d0ugal: so we can merge patches which fails on those gates? | 12:34 |
| d0ugal | oidgar: I have been :) | 12:34 |
| oidgar | d0ugal: great, thanks! | 12:34 |
| hjensas | oidgar: we do, but I am on a train today. Network/cell coverage is not sufficent for me to join. | 12:35 |
| oidgar | hjensas: I don't think anyone else is joining so probably it is canceled. thanks | 12:35 |
| d0ugal | No meetings should be allowed this week :) | 12:37 |
| caboucha | I too have been noticing failures in gate | 12:41 |
| caboucha | first time committing to tripleo | 12:41 |
| caboucha | https://review.openstack.org/527508 | 12:42 |
| *** psahoo has quit IRC | 12:42 | |
| caboucha | I'll keep looking to see if it's me | 12:42 |
| *** dpawar has joined #tripleo | 12:51 | |
| Tengu | owalsh: I'm back, just saw your review request. Will check that shortly :). | 12:52 |
| Tengu | hmmm.... gerrit is slow as hell. | 12:52 |
| *** pgadiya has quit IRC | 12:53 | |
| *** yolanda__ is now known as yolanda | 12:53 | |
| *** jpena is now known as jpena|lunch | 12:58 | |
| *** dmellado has quit IRC | 13:05 | |
| *** dprince has joined #tripleo | 13:08 | |
| Tengu | gerrit is dead, apparently. | 13:08 |
| Tengu | getting 502 errors. | 13:08 |
| Tengu | oidgar: d0ugal can confirm: 3rd party isn't stable nor reliable for CI - already seen constent failure for a working code (due to timeouts or such non-code related) | 13:10 |
| oidgar | Tengu: thanks | 13:10 |
| d0ugal | Thanks Tengu | 13:10 |
| Tengu | and I was told by others "nah, don't care". or things like that ;) | 13:11 |
| Tengu | digging in the CI logs is a painful exercise. | 13:11 |
| *** abregman has quit IRC | 13:11 | |
| *** openstackgerrit has quit IRC | 13:13 | |
| *** fpan has joined #tripleo | 13:14 | |
| Tengu | ah. according to the ML, gerrit is under maintenance due to some issue. | 13:15 |
| *** dmellado has joined #tripleo | 13:15 | |
| -openstackstatus- NOTICE: gerrit is being restarted due to extreme slowness | 13:15 | |
| Tengu | voilà :) | 13:15 |
| *** amoralej is now known as amoralej|lunch | 13:16 | |
| *** openstackgerrit has joined #tripleo | 13:17 | |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-heat-templates stable/pike: Check for yum lock befor all yum* operations. https://review.openstack.org/529309 | 13:17 |
| *** stevebaker has quit IRC | 13:18 | |
| jaganathan | d0ugal, please look into https://review.openstack.org/#/c/522265/ | 13:19 |
| *** dmellado has quit IRC | 13:19 | |
| *** hewbrocca_afk is now known as hewbrocca | 13:20 | |
| *** dmellado has joined #tripleo | 13:21 | |
| *** pdeore has joined #tripleo | 13:22 | |
| *** jaganathan has quit IRC | 13:23 | |
| *** rlandy has joined #tripleo | 13:23 | |
| *** rlandy is now known as rlandy|ruck | 13:24 | |
| openstackgerrit | Martin André proposed openstack/tripleo-quickstart-extras master: DNM: Update quickstart extras with undercloud install for containers https://review.openstack.org/517444 | 13:26 |
| *** abregman has joined #tripleo | 13:27 | |
| *** dmacpher has joined #tripleo | 13:27 | |
| *** stevebaker has joined #tripleo | 13:28 | |
| *** lucas-hungry is now known as lucasagomes | 13:28 | |
| *** BryanS68 has joined #tripleo | 13:32 | |
| *** rmascena has joined #tripleo | 13:35 | |
| *** raildo has quit IRC | 13:37 | |
| *** pchavva has joined #tripleo | 13:39 | |
| *** rhallisey has quit IRC | 13:40 | |
| *** skramaja has quit IRC | 13:40 | |
| *** jcoufal has joined #tripleo | 13:43 | |
| *** rhallisey has joined #tripleo | 13:43 | |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Add docker-registry service https://review.openstack.org/526132 | 13:45 |
| weshay | mwhahaha, morning.. ping me when you have a sec re: container updates | 13:46 |
| weshay | I gave it a go w/ puppet-nova but I need to verify the results | 13:47 |
| *** dpawar has quit IRC | 13:48 | |
| *** trown|outtypewww is now known as trown | 13:48 | |
| openstackgerrit | Sven Anderson proposed openstack/tripleo-quickstart master: TEST DON'T MERGE - Enabling EC2-API Tempest tests. https://review.openstack.org/515139 | 13:49 |
| *** psachin has quit IRC | 13:49 | |
| *** amoralej|lunch is now known as amoralej | 13:50 | |
| openstackgerrit | Martin André proposed openstack/tripleo-quickstart-extras master: DNM: Update quickstart extras with undercloud install for containers https://review.openstack.org/517444 | 13:51 |
| *** rlandy|ruck is now known as rlandy|rover | 13:52 | |
| *** catintheroof has joined #tripleo | 13:52 | |
| *** trown is now known as trown|ruck | 13:52 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-docs master: Document using Ironic Ansible deploy interface https://review.openstack.org/526663 | 13:53 |
| *** jpena|lunch is now known as jpena | 13:54 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart-extras master: Add ansible update into UpgradeInitCommand of repo template https://review.openstack.org/528261 | 13:58 |
| *** dpawar has joined #tripleo | 13:58 | |
| *** bregman has joined #tripleo | 13:58 | |
| *** rbowen has joined #tripleo | 14:00 | |
| *** bfournie has joined #tripleo | 14:00 | |
| *** agopi has joined #tripleo | 14:01 | |
| *** jwb has joined #tripleo | 14:01 | |
| *** abregman has quit IRC | 14:02 | |
| *** catintheroof has quit IRC | 14:02 | |
| dtantsur | ansiwen: o/ up for questions re real time virt? | 14:03 |
| *** jmelvin has joined #tripleo | 14:04 | |
| *** catintheroof has joined #tripleo | 14:04 | |
| *** yprokule has quit IRC | 14:04 | |
| *** yprokule has joined #tripleo | 14:05 | |
| *** liverpooler has joined #tripleo | 14:07 | |
| Tengu | owalsh: are you still here? any knowledge about node deletion? | 14:08 |
| *** pdeore has quit IRC | 14:10 | |
| *** agurenko has quit IRC | 14:11 | |
| Tengu | erf… doc for node removal is also deprecated X( | 14:12 |
| *** agopi has quit IRC | 14:14 | |
| alee | mwhahaha, EmilienM , rlandy|rover https://review.openstack.org/#/c/527136/ failed to get off the ground -- merge conflict somewhere | 14:18 |
| owalsh | Tengu: don't know much about it, I've run it once or twice maybe | 14:19 |
| Tengu | owalsh: ok. running one right now, got a timeout with some websocket, but apparently the removal is running | 14:20 |
| Tengu | stack is updated. | 14:20 |
| Tengu | but the --help for `openstack overcloud node delete' was a bit confusing, when we read the doc in // | 14:21 |
| Tengu | makes me say: doc isn't up-to-date and might create some issues shortly. | 14:21 |
| *** catinthe_ has joined #tripleo | 14:22 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-common master: Parameterize ceph-ansible forks in Mistral Workflow https://review.openstack.org/528124 | 14:23 |
| openstackgerrit | Merged openstack/instack-undercloud master: Load undercloud DB password to a mistral environment https://review.openstack.org/518292 | 14:24 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Convert tags to when statements for Q major upgrade workflow https://review.openstack.org/510902 | 14:25 |
| openstackgerrit | John Fulton proposed openstack/tripleo-common master: Parameterize ceph-ansible forks in Mistral Workflow https://review.openstack.org/528124 | 14:25 |
| *** catintheroof has quit IRC | 14:26 | |
| rlandy|rover | alee: lookinh | 14:27 |
| rlandy|rover | looking | 14:27 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-quickstart-extras master: Make suer we use quickstart extras from the $WORKSPACE https://review.openstack.org/529334 | 14:27 |
| openstackgerrit | John Fulton proposed openstack/tripleo-common master: Parameterize ceph-ansible forks in Mistral Workflow https://review.openstack.org/528124 | 14:28 |
| *** dciabrin has quit IRC | 14:29 | |
| rlandy|rover | alee: where did you see a merge conflict? | 14:30 |
| *** dciabrin has joined #tripleo | 14:30 | |
| alee | rlandy|rover, last comment for zuul in https://review.openstack.org/#/c/527136/ | 14:30 |
| alee | rlandy|rover, I'm not sure where it happens .. | 14:31 |
| EmilienM | alee: because your patches in Depends-On were updated | 14:31 |
| EmilienM | so Zuul asks you to recheck | 14:31 |
| alee | EmilienM, ah ok - rechecking | 14:32 |
| EmilienM | Tengu: I was sleeping :-) - have you found on https://docs.openstack.org/tripleo-docs/latest/ ? | 14:33 |
| *** openstackgerrit has quit IRC | 14:33 | |
| Tengu | EmilienM: wow, you had a long night then ;) | 14:33 |
| Tengu | EmilienM: https://docs.openstack.org/tripleo-docs/latest/install/post_deployment/delete_nodes.html yup, and apparently, according the the --help, the "-e" isn't needed anymore and is deprecated. | 14:33 |
| *** openstackgerrit has joined #tripleo | 14:35 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud master: Generate a temporary URL key for Swift "service" project https://review.openstack.org/527376 | 14:35 |
| *** oidgar has quit IRC | 14:35 | |
| *** lblanchard has joined #tripleo | 14:37 | |
| *** trown|ruck is now known as trown|brb | 14:39 | |
| *** shardy has joined #tripleo | 14:41 | |
| *** ykarel has quit IRC | 14:43 | |
| *** trown|brb is now known as trown | 14:47 | |
| *** oidgar has joined #tripleo | 14:48 | |
| Tengu | hmmm. node deletion seems to be stuck in a sub-stack, named overcloud-ComputeSshKnownHostsDeployment-3ffgrpdxnrmw - its name seems to indicate it's just the know host in /etc/ssh and, probably, the /etc/hosts file edition… | 14:50 |
| *** sshnaidm has quit IRC | 14:50 | |
| mwhahaha | weshay: whats up | 14:50 |
| *** sshnaidm has joined #tripleo | 14:50 | |
| mwhahaha | alee: ah blame weshay for messing with the dependencies | 14:51 |
| *** trown is now known as trown|ruck | 14:52 | |
| *** shreshtha has joined #tripleo | 14:53 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: Enable support for ironic "direct" deploy interface https://review.openstack.org/529342 | 14:53 |
| * weshay looking at https://review.openstack.org/#/c/515372/ that has a dep on puppet-nova https://review.openstack.org/#/c/529183/ when I look at the rpms getting updated I don't see it. Wondering if I'm correct in understanding that I should see it on a container.. but maybe not | 14:53 | |
| weshay | http://logs.openstack.org/72/515372/16/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e0d7055/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz | 14:53 |
| *** lblanchard has quit IRC | 14:54 | |
| weshay | http://logs.openstack.org/72/515372/16/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e0d7055/logs/subnode-2/var/log/yum.log.txt.gz | 14:54 |
| mwhahaha | puppet-nova-12.1.1-0.20171220092840.241e81c.el7.centos.noarch | 14:54 |
| mwhahaha | that's updated | 14:54 |
| weshay | ya.. on the host, not the container | 14:56 |
| *** bregman has quit IRC | 14:56 | |
| weshay | do you have a suggestion of a repo I can dep on? | 14:56 |
| weshay | to see the update in the containers | 14:56 |
| mwhahaha | well alee's patch would be that | 14:57 |
| mwhahaha | since he's deping on nova | 14:57 |
| weshay | his patch has a long train of deps | 14:57 |
| weshay | so just openstack/nova | 14:57 |
| mwhahaha | to check the containers you'd need a normal openstack project | 14:57 |
| weshay | makes sense | 14:57 |
| weshay | ya | 14:57 |
| weshay | k k | 14:57 |
| weshay | rlandy|rover, fyi ^ | 14:57 |
| mandre | weshay: let me know when you have a minute to look at the containerized undercloud patch with me | 14:57 |
| weshay | sshnaidm, ^ | 14:58 |
| weshay | mandre, ok.. give me 5min | 14:58 |
| mandre | weshay: cool | 14:58 |
| mwhahaha | mandre: i heard rumblings that the containerized undercloud was going to be a requirement for FFU, do you know if that's the case and why? | 14:58 |
| shardy | Anyone know how to monitor the status of promotion for the https://trunk.rdoproject.org/centos7/current/ pin? It seems that's no longer actually trunk, so there's a period after a patch lands where we're using old packages and CI jobs fail where a Depends-On exists | 14:59 |
| rlandy|rover | weshay: thanks for following this up | 14:59 |
| mandre | mwhahaha: from what I understood, it's not a strict requirement for FFU but it would make it more maintainable | 14:59 |
| mwhahaha | mandre: ok. i'm not sure i get that maintainable claim | 15:00 |
| *** catinthe_ has quit IRC | 15:02 | |
| *** ykarel has joined #tripleo | 15:02 | |
| dtantsur | I've heard this too, and I'm not sure why baremetal->baremetal is harder than baremetal->containers either | 15:02 |
| *** catintheroof has joined #tripleo | 15:03 | |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-quickstart-extras master: Use openstack commands in overcloud-deploy.sh https://review.openstack.org/529347 | 15:03 |
| tbarron | bfournie: do you have a view on https://review.openstack.org/#/c/523638/ ? is it close to merger or not? | 15:03 |
| tbarron | bfournie: I'm lining up all the unresolved dependencies for our manila ceph-nfs work in light of potential Feature Freeze Exceptions, etc. and this is a big one. | 15:04 |
| bfournie | tbarron: some of my comments from 4 still aren't addressed, I think Dan's still working on change for the management network, I think its really close but Dan will no better when he's online | 15:05 |
| bfournie | s/no/know | 15:05 |
| tbarron | bfournie: k, I'll ask both of you again when the west coast wakes up. Thanks. | 15:06 |
| tbarron | hjensas: ^^ I see you on that review too, and Welcome! | 15:06 |
| *** catintheroof has quit IRC | 15:08 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: DNM: test container updates https://review.openstack.org/515372 | 15:08 |
| *** nyechiel has quit IRC | 15:10 | |
| weshay | mandre, hey | 15:11 |
| *** shardy has quit IRC | 15:11 | |
| mandre | hey weshay, so I was looking at https://review.openstack.org/#/c/517444/ and was wondering what was the reason for only keeping localhost in /etc/hosts | 15:12 |
| weshay | mandre, so initially dprince found duplicate entries in hosts and that was causing issues w/ rabbit according to eck`. So I first removed the duplicates, and the undercloud was still failing to deploy. | 15:13 |
| weshay | I next checked the overcloud node hosts file and noticed it ONLY had localhost, so figuring the containerized undercloud is more like the previous overcloud deployment. Also locally it was working for me and noticed the hosts file only had localhost | 15:14 |
| weshay | once we purged the hosts file from what infra put in .. it started working and completing the undercloud deployment | 15:15 |
| mandre | ok, do you mind me removing this? I'm thinking this is messing up with CI | 15:15 |
| weshay | mandre, give it a go, but I don't think it will work | 15:15 |
| weshay | it least it hasn't in the past, maybe something changed | 15:15 |
| mandre | weshay: or do you have an idea why almost all the jobs are red at https://review.openstack.org/#/c/517444/? | 15:16 |
| *** sshnaidm is now known as sshnaidm|afk | 15:16 | |
| mandre | http://logs.openstack.org/44/517444/40/check/tripleo-ci-centos-7-containers-multinode/56e2191/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 15:17 |
| weshay | http://logs.openstack.org/44/517444/40/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/baeef16/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 15:18 |
| weshay | heh | 15:18 |
| weshay | ok.. let's take it out and see what happens | 15:18 |
| *** oidgar has quit IRC | 15:20 | |
| openstackgerrit | Martin André proposed openstack/tripleo-quickstart-extras master: DNM: Update quickstart extras with undercloud install for containers https://review.openstack.org/517444 | 15:20 |
| *** myoung is now known as myoung|bbl | 15:20 | |
| mandre | weshay: we'll know in 30 min ^^ | 15:20 |
| *** shardy has joined #tripleo | 15:24 | |
| *** masco has quit IRC | 15:29 | |
| *** karthiks has quit IRC | 15:32 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Fix reproducer script path references for all environments https://review.openstack.org/529356 | 15:32 |
| *** dpawar has quit IRC | 15:36 | |
| *** fragatina has quit IRC | 15:37 | |
| *** fragatina has joined #tripleo | 15:37 | |
| *** moshele has quit IRC | 15:38 | |
| *** trozet has quit IRC | 15:40 | |
| *** trozet has joined #tripleo | 15:43 | |
| rook | shardy: so, * 2 might not be a bad idea | 15:46 |
| rook | shardy had 12, and it was soo slow. | 15:47 |
| *** hjensas has quit IRC | 15:47 | |
| *** jongwooh has quit IRC | 15:47 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-common master: Parameterize Ansible environment vars in Mistral Workflow https://review.openstack.org/528124 | 15:51 |
| *** pcaruana has joined #tripleo | 15:51 | |
| *** janki has quit IRC | 15:53 | |
| ansiwen | dtantsur: sorry, misse you. still around? now I'm up to it :-) | 15:53 |
| shardy | rook: Ack Ok, would you mind commenting on https://review.openstack.org/#/c/529066/ and/or the bug so we can track your results and agree a reasonable default? | 15:57 |
| shardy | rook: thanks for the update! | 15:57 |
| dtantsur | ansiwen: hey! I got some good progress with the ansible deploy. I managed to set custom kernel params, see doc https://review.openstack.org/526663 | 15:58 |
| dtantsur | ansiwen: now I'm going through your google doc, trying to figure out how it all maps to this work | 15:58 |
| openstackgerrit | Mark Hamzy proposed openstack/tripleo-common master: [WIP] Support multiple architectures https://review.openstack.org/528000 | 15:58 |
| dtantsur | ansiwen: so, question #1: why not pre-install packages on the overcloud-full image? | 15:58 |
| *** moshele has joined #tripleo | 16:01 | |
| *** jongwooh has joined #tripleo | 16:02 | |
| *** jcoufal has quit IRC | 16:04 | |
| *** BryanS68 has quit IRC | 16:07 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: Parameterize ceph-ansible environment variables https://review.openstack.org/528125 | 16:07 |
| *** ykarel has quit IRC | 16:09 | |
| *** ykarel has joined #tripleo | 16:09 | |
| *** ykarel has quit IRC | 16:11 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud master: Generate a temporary URL key for Swift "service" project https://review.openstack.org/527376 | 16:15 |
| *** jcoufal has joined #tripleo | 16:24 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Correct links for images https://review.openstack.org/516353 | 16:25 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Check for yum lock befor all yum* operations. https://review.openstack.org/528984 | 16:25 |
| *** moshele has quit IRC | 16:29 | |
| ansiwen | dtantsur: sorry, just got a phone call. so: the kernel you can't install on the overcloud image, because it replaces the standard kernel. | 16:30 |
| dtantsur | ansiwen: ugh. so, setting up repositories and installing packages is not impossible, but assumes access to the internet during deployment | 16:31 |
| *** marrusl has quit IRC | 16:33 | |
| openstackgerrit | Merged openstack/tripleo-common master: SRIOV derive parameters workflows https://review.openstack.org/522265 | 16:34 |
| *** rbrady is now known as rbrady-afk | 16:36 | |
| *** Goneri has joined #tripleo | 16:37 | |
| owalsh | ansiwen, dtantsur: can have both kernel installed IIRC | 16:39 |
| dtantsur | worth figuring out IMO | 16:40 |
| EmilienM | shardy, rook : any thoughts on https://review.openstack.org/#/c/529130/ ? | 16:40 |
| ansiwen | dtantsur: but this requirement is also given for the cirrent script in the doc I sent you, right? so I think that wouldn't be a "regression" | 16:40 |
| dtantsur | ansiwen: well, I'm trying to figure out what it takes to move your script to an ironic ansible playbook | 16:41 |
| ansiwen | owalsh: ok, interesting? how to install it without "enable" it? and how to enable it afterwards? over grub default? | 16:41 |
| dtantsur | e.g. networking during deploy does not have to be set up the same way as on the final instance (incl. during first boot) | 16:41 |
| dtantsur | i.e. IPA does not have to be able to access internet | 16:41 |
| dtantsur | which will complicate fetching packages | 16:41 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Fix nodes config path in reproducer script https://review.openstack.org/529367 | 16:43 |
| sshnaidm|afk | trown|ruck, rlandy|rover ^^ | 16:43 |
| trown|ruck | sshnaidm|afk: rlandy|rover has a similar review | 16:44 |
| trown|ruck | https://review.openstack.org/#/c/529356/1 | 16:44 |
| *** links has quit IRC | 16:44 | |
| rlandy|rover | sshnaidm|afk: trown|ruck: already addressed with other changes in https://review.openstack.org/#/c/529356/ | 16:44 |
| rlandy|rover | sshnaidm|afk: adding you to that review | 16:44 |
| rlandy|rover | feel free to modify but let;s keep one to avoid merge issues | 16:45 |
| sshnaidm|afk | oops | 16:45 |
| rlandy|rover | np - my fault for not adding you | 16:46 |
| *** lucasagomes is now known as lucas-afk | 16:48 | |
| *** moshele has joined #tripleo | 16:48 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-common master: Remove step_tags_to_when function from config download https://review.openstack.org/529369 | 16:48 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: Parameterize ceph-ansible environment variables https://review.openstack.org/528125 | 16:50 |
| openstackgerrit | Sayali Lunkad proposed openstack/diskimage-builder master: Adding mapping for SUSE package https://review.openstack.org/529370 | 16:51 |
| *** moshele has quit IRC | 16:51 | |
| *** cshastri has joined #tripleo | 16:53 | |
| *** jtomasek has quit IRC | 16:54 | |
| ansiwen | dtantsur: I see... the script is run in different context than the ansible playbook. so I will check all the packages. if we can all add them to the overcloud image in parallel without risks for the current roles, sure, let's do that. otherwise it will be hard to add a risk so late in the cycle. | 16:56 |
| *** marios has quit IRC | 16:57 | |
| dtantsur | ansiwen: yep, let's check it first | 16:57 |
| dtantsur | ansiwen: are you coming to the office tomorrow? I'll have to run soon today, but we can chat face-to-face | 16:57 |
| ansiwen | dtantsur: you come to the brerakfast? ok, cool! let's do that, that'd be great! | 16:58 |
| *** anilvenkata has quit IRC | 16:58 | |
| dtantsur | :) | 16:58 |
| dtantsur | if I figure out how to enter the office this time... | 16:58 |
| Tengu | anyone can help me in order to clean a failed resource in tripleo "overcloud" heat stack? the failed resource is: overcloud-ComputeSshKnownHostsDeployment-3ffgrpdxnrmw - I don't know why this #@|@#¼ tasks is failing, it's supposed to clean removed compute nodes, but is always failing due to some timeout. | 16:59 |
| Tengu | it's starting to really annoy me, as it prevents any stack update, like adding new node -.- | 16:59 |
| *** ykarel has joined #tripleo | 17:01 | |
| ansiwen | dtantsur: go to 3rd floor first and take the stairs to 2nd and knock loudly on the door. (it's probably on 2nd floor this time) | 17:02 |
| dtantsur | ansiwen: what if I telegram you when I approach the building, so that you meet me on the 3rd floor? | 17:15 |
| *** mdnadeem has quit IRC | 17:17 | |
| ansiwen | dtantsur: you can try, but I'm often too late. :-) but the receptionist can escort you to the breakfast in any case. :-) | 17:17 |
| dtantsur | assuming they know where it is.. | 17:18 |
| ansiwen | dtantsur: they set it up, so they _must_ know it :-) | 17:18 |
| *** udesale has quit IRC | 17:18 | |
| dtantsur | cool :) | 17:19 |
| * dtantsur gets late coffee now | 17:19 | |
| *** florianf has quit IRC | 17:19 | |
| *** cshastri has quit IRC | 17:20 | |
| *** ykarel has quit IRC | 17:21 | |
| *** dprince has quit IRC | 17:23 | |
| *** d0ugal has quit IRC | 17:23 | |
| *** trown|ruck is now known as trown|lunch | 17:25 | |
| *** gkadam has quit IRC | 17:27 | |
| *** salmankhan has quit IRC | 17:29 | |
| *** jfrancoa has quit IRC | 17:29 | |
| *** hewbrocca is now known as hewbrocca_afk | 17:30 | |
| *** dtantsur is now known as dtantsur|afk | 17:34 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-puppet-elements master: WIP: add RT kernel to overcloud compute image https://review.openstack.org/529381 | 17:34 |
| owalsh | dtantsur|afk, ansiwen: ^^^ that should work I think, installs the RT kernel but restores the default back to the non-RT kernel | 17:34 |
| dtantsur|afk | thanks | 17:35 |
| * dtantsur|afk goes now | 17:35 | |
| *** d0ugal has joined #tripleo | 17:35 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-puppet-elements master: WIP: add RT kernel to overcloud compute image https://review.openstack.org/529381 | 17:36 |
| EmilienM | shardy: would you mind to review https://review.openstack.org/#/c/526151/ please? | 17:37 |
| *** salmankhan has joined #tripleo | 17:37 | |
| Tengu | what would happen if I comment out this bloc and deploy/update the overcloud stack? https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/overcloud.j2.yaml#L450-L455 | 17:41 |
| Tengu | I think it should drop 3 stacks - and when I uncomment it, it should create it back. Is that right and safe? | 17:42 |
| *** marrusl has joined #tripleo | 17:42 | |
| Tengu | or, shall I flag the stack as "failed" (openstack stack resource mark unhealthy <resource>) and re-deploy? | 17:43 |
| *** etingof has quit IRC | 17:43 | |
| *** fultonj has quit IRC | 17:47 | |
| Tengu | hmmm. apparently, marking the nested stack as unhealthy should do the trick. | 17:47 |
| *** derekh has quit IRC | 17:52 | |
| rook | EmilienM: I mentioned to Shardy that 12 might not be enough workers (very simplistic findings from trying 12 workers with a 90 node deployment) | 17:53 |
| owalsh | Tengu: ah, so that's why you're using StrictHostKeyChecking no | 17:53 |
| owalsh | Tengu: why do you need to disabled it? | 17:53 |
| EmilienM | rook: what's the right formula for you then? | 17:53 |
| rook | EmilienM: shardy mentioned the calculation you had *2. | 17:54 |
| rook | so, if you default to the max of 12, it would be 24 | 17:54 |
| EmilienM | ok | 17:54 |
| rook | EmilienM: which I have tested and that does work much better. | 17:54 |
| rook | However, we are around 1GB per worker :/ | 17:55 |
| rook | so, back to the memory consumption issue | 17:55 |
| ansiwen | owalsh: oh, cool, thanks! | 17:55 |
| *** pchavva has quit IRC | 17:55 | |
| EmilienM | rook: is https://review.openstack.org/529130 better onw? | 17:56 |
| EmilienM | now* | 17:56 |
| rook | https://snapshot.raintank.io/dashboard/snapshot/zQODzQetB56fGgDahLLAAu2zifpB11Bx?orgId=2 <-- showing the usage | 17:56 |
| tdasilva | mwhahaha, EmilienM looking for some help regarding swift+barbican integration. alee has made the change to install barbican in step3 but now I need to add a little script to create a secret and stick the key_id in the swift conf file. Where should that script be executed? | 17:58 |
| *** yprokule has quit IRC | 18:00 | |
| Tengu | owalsh: ah, well, no, nothing to do with that. unrelated :) | 18:00 |
| Tengu | owalsh: fact is, tripleo deploy process is failing on that precise task, probably due to some error I made earlier. And I can't manage to recover :( | 18:01 |
| Tengu | owalsh: so I'm reading and thinking of a way to sort that situation. I see two possibilities: either mark the specific resource as "unhealthy" in heat, or drop that particular thing from the overcloud.j2 and re-deploy so that it should drop it. | 18:02 |
| Tengu | owalsh: fact is: I think marking it as unhealthy should be the right thing. | 18:02 |
| mwhahaha | tdasilva: so i assume it needs to go into the appropriate place in docker_config in step4+ | 18:02 |
| owalsh | Tengu: any custom roles? | 18:03 |
| *** dprince has joined #tripleo | 18:03 | |
| Tengu | owalsh: nope | 18:03 |
| EmilienM | rook: please give feedback on https://review.openstack.org/#/c/529130/ | 18:03 |
| Tengu | owalsh: basic ones, but there's a name mapping, and I failed it, and it kind of messed up the compute-related stacks. | 18:03 |
| *** dsneddon has joined #tripleo | 18:04 | |
| *** fultonj has joined #tripleo | 18:05 | |
| owalsh | Tengu: ok, I've not seen any issues with the ssh known hosts setup, but there's always a first time | 18:06 |
| Tengu | owalsh: :) | 18:07 |
| Tengu | owalsh: so, marking a task as unhealthy should replace it "in-place" right? | 18:07 |
| owalsh | Tengu: no idea, shardy? | 18:08 |
| Tengu | owalsh: in my case, I have the stack name, it's precisely overcloud-ComputeSshKnownHostsDeployment-3ffgrpdxnrmw, and it has only one resource. Reading the overcloud.j2 makes me think it's only managing ssh known host for computes, so it should NOT impact anything else. | 18:09 |
| tdasilva | mwhahaha: but won't that be executed in every controller node? my thought is that it would be executed once and then stick the result in here: https://review.openstack.org/#/c/525324/2/puppet/services/swift-proxy.yaml@154 | 18:11 |
| tdasilva | using get_param | 18:11 |
| mwhahaha | tdasilva: the problem is that under containerization the puppet stuff may not be run at the same time | 18:11 |
| owalsh | Tengu: all hosts, it's in a {% for role in roles %} loop | 18:12 |
| mwhahaha | tdasilva: I think there's a way to run the docker_tasks only on a bootstrap node | 18:12 |
| mwhahaha | tdasilva: this is the problem with needing dynamic config items that are not generated prior to deployment | 18:12 |
| Tengu | owalsh: hmm... so it will deploy the compute nodes keys on the other roles? anyway, it's only ssh keys... | 18:12 |
| rook | EmilienM: ack | 18:12 |
| tdasilva | mwhahaha: i heard you :/ | 18:12 |
| Tengu | I guess I can mark it unhealty… :/ | 18:12 |
| tdasilva | i hear you | 18:12 |
| tdasilva | mwhahaha: recently learned that cinder is creating these keys per tenant and wished we had done the same | 18:13 |
| owalsh | Tengu: yea, combines all of the ssh host keys and generates /etc/ssh/ssh_known_hosts on all hosts (so we don't need StrictHostKeyChecking no) | 18:13 |
| * owalsh biab | 18:13 | |
| Tengu | owalsh: ok. so I don't really have risk marking it. hopefully. | 18:14 |
| rook | fultonj: the last patch you applied set delegate_facts to false, right? | 18:14 |
| rook | Seb is asking me. | 18:14 |
| Tengu | shardy: are you here? :) | 18:14 |
| mwhahaha | tdasilva: so the setting of config item itself needs to on all nodes, it's jsut teh single generation of the key that you'd need to do once right? | 18:14 |
| fultonj | rook: yes | 18:14 |
| fultonj | i harded coded that | 18:14 |
| rook | ok | 18:14 |
| rook | I didn't see it hard coded. | 18:14 |
| rook | i looked int he ceph-ansible dir. | 18:14 |
| *** etingof has joined #tripleo | 18:14 | |
| rook | in the* | 18:14 |
| tdasilva | mwhahaha: correct | 18:15 |
| mwhahaha | owalsh: how do we prevent the bootstrap bits for nova from being run on multiple nodes | 18:15 |
| mwhahaha | owalsh: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/nova-api.yaml#L193 what i was looking at | 18:15 |
| fultonj | rook: http://ix.io/Dfc | 18:15 |
| rook | I see it here fultonj site-docker.yml.sample | 18:15 |
| fultonj | ^ yep | 18:15 |
| rook | ok | 18:15 |
| rook | cool, then we are on the same page. | 18:16 |
| rook | becuase i see it set to true here : infrastructure-playbooks/rolling_update.yml | 18:16 |
| fultonj | that playbook isn't used on deploy though | 18:16 |
| rook | nope, that shouldn't be in question | 18:16 |
| rook | however, that would override a configuration | 18:17 |
| *** yamahata has joined #tripleo | 18:17 | |
| mwhahaha | tdasilva: so i'm not exactly sure the best place to drop the items as it relates to containers so might be a good idea to ask the containers folks. I think you'll probably need a docker_config item but not sure the best way to implement your script | 18:20 |
| tdasilva | mwhahaha: no worries, i'll ping the containers folks, thanks! | 18:22 |
| *** etingof has quit IRC | 18:22 | |
| *** eck` is now known as eck`gone | 18:25 | |
| fultonj | rook: i replied to seb | 18:25 |
| rook | ok fultonj | 18:26 |
| * rook closes window | 18:26 | |
| rook | fultonj: where is the best place to chat with seb live? | 18:26 |
| fultonj | rook: i will pm you | 18:27 |
| *** eck`gone is now known as eck` | 18:30 | |
| *** rhallisey has quit IRC | 18:31 | |
| *** etingof has joined #tripleo | 18:35 | |
| fultonj | rook: did you get to run your change with https://gist.github.com/jtaleric/e8c3f6f6137751ab89e20efd8093643b ? | 18:37 |
| rook | it is running now. | 18:38 |
| fultonj | ack | 18:38 |
| openstackgerrit | Sven Anderson proposed openstack/tripleo-quickstart master: TEST DON'T MERGE - Enabling EC2-API Tempest tests. https://review.openstack.org/515139 | 18:39 |
| *** pchavva has joined #tripleo | 18:39 | |
| *** jtomasek has joined #tripleo | 18:40 | |
| *** salmankhan has quit IRC | 18:41 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Remove MTU-based tests from the master and pike skip lists https://review.openstack.org/528292 | 18:42 |
| weshay | mwhahaha, k.. this is working.. thanks for the help https://review.openstack.org/#/c/501028/ | 18:46 |
| weshay | it's off by default atm, we'll send a patch to turn it on upstream | 18:46 |
| weshay | rlandy|rover, fyi ^ | 18:47 |
| mwhahaha | k | 18:47 |
| weshay | Slower++ | 18:47 |
| *** rbrady-afk is now known as rbrady | 18:48 | |
| rlandy|rover | finally :) - started September 15 | 18:48 |
| * mwhahaha points out container-check should probably be packaged | 18:48 | |
| owalsh | mwhahaha: bootstrap_host_exec https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/nova-api.yaml#L206 | 18:48 |
| mwhahaha | owalsh: ah so that ensure it only runs on the bootstrap node | 18:49 |
| mwhahaha | totally not obvious :D | 18:49 |
| *** oidgar has joined #tripleo | 18:50 | |
| owalsh | mwhahaha: :-) yea, noop if the hostnames don't match hiera IIRC | 18:51 |
| *** ebarrera has quit IRC | 18:51 | |
| mwhahaha | tdasilva: so if you want to run a shell script on a single node you can do so witht he bootstrap_host_exec -^ but the actual config writing out would be hard to do on all systems without publishing that key somewhere. So maybe you just need a script that runs everywhere and includes the bootstrap check | 18:51 |
| *** ebarrera has joined #tripleo | 18:53 | |
| *** oidgar has quit IRC | 18:55 | |
| *** ebarrera_ has joined #tripleo | 19:01 | |
| *** myoung|bbl is now known as myoung | 19:01 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Update containers when the overcloud is containerized https://review.openstack.org/529399 | 19:04 |
| rook | https://gist.github.com/jtaleric/4dee30154651ecdedc79ea820a0d3c10 fultonj have you seen this before? | 19:05 |
| rook | Or anyone... | 19:05 |
| rlandy|rover | weshay: ^^ update_containers settings in the testenv files | 19:05 |
| rook | The gist of the gist... Overcloud deployment is in flight. Node gets beyond build... Node never reboots (from ironic)... So, pinging the IP fails, Deployment gets hung up. The only way to progress the deployment is to do what I dd in the gist. | 19:06 |
| rook | Shut down the trouble node, and start it back up. | 19:06 |
| rook | this used to happen a lot more frequently, which led to people having to baby-sit deployments. | 19:07 |
| fultonj | rook: i have seen that occasionally | 19:07 |
| rook | It has happened 4x in this scale deployment. | 19:07 |
| rook | I will admit, I haven't seen it for a while. | 19:07 |
| *** trown|lunch is now known as trown|brb | 19:08 | |
| *** trown|brb is now known as trown | 19:08 | |
| *** trown is now known as trown|ruck | 19:08 | |
| *** ebarrera_ has quit IRC | 19:10 | |
| *** oidgar has joined #tripleo | 19:11 | |
| *** oidgar has quit IRC | 19:13 | |
| *** holser__ has quit IRC | 19:14 | |
| rook | fultonj: this seems like a expensive operation : 2017-12-20 19:11:58,144 p=356309 u=mistral | TASK [ceph-defaults : set_fact fsid ceph_current_fsid.stdout] ****************** | 19:14 |
| weshay | need a third party vote on https://review.openstack.org/#/c/509660/ | 19:19 |
| weshay | rlandy|rover, ^ | 19:19 |
| weshay | EmilienM, do you have a sec? | 19:19 |
| rlandy|rover | conflict of interest | 19:19 |
| rlandy|rover | my code | 19:19 |
| EmilienM | weshay: of course | 19:19 |
| weshay | thank you sir | 19:19 |
| rook | rlandy|rover you should of skipped that ethics training. | 19:19 |
| EmilienM | chem: https://review.rdoproject.org/r/#/c/10827/ needs rebase fyi | 19:20 |
| rlandy|rover | rook: lol - the ethics training didn't cover w+1'ing your own code - it should | 19:20 |
| EmilienM | weshay: too late trown|ruck approved it :) but lgtm as well | 19:21 |
| weshay | aight | 19:23 |
| weshay | thanks anyway | 19:23 |
| EmilienM | exciting times, tripleo periodic jobs run again | 19:23 |
| EmilienM | have they already run today? | 19:23 |
| *** shardy has quit IRC | 19:23 | |
| EmilienM | weshay: re: https://review.openstack.org/#/c/526138/ - thanks again for this work, I would send an email to the ML + a patch in tripleo-docs to announce this cool feature. Thanks | 19:23 |
| trown|ruck | EmilienM: yes... though not with passing results | 19:24 |
| tdasilva | mwhahaha: one way i was thinking about doing is having one exec to create the key, and then a second exec to get the key could be run in all nodes, the problem is that I need to pass an uuid between the create and the get | 19:25 |
| EmilienM | trown|ruck: what were the failures? | 19:27 |
| weshay | EmilienM, ok.. I'll look at tripleo-docs with a more holistic view with regards to ci again in a bit | 19:28 |
| trown|ruck | EmilienM: i havent got to all of the pike ones yet... master all the code passed, but there is some issue with the qcow image upload, working on getting a bug for that, then looking at pike | 19:28 |
| EmilienM | trown|ruck: I can help you | 19:28 |
| *** pcaruana has quit IRC | 19:29 | |
| EmilienM | trown|ruck: if you have a link for the pike ones, I can take a look now | 19:29 |
| *** oidgar has joined #tripleo | 19:29 | |
| trown|ruck | https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-master-upload/417/console | 19:29 |
| trown|ruck | https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-pike/ | 19:29 |
| trown|ruck | https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/ | 19:29 |
| trown|ruck | https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-pike-upload/ | 19:29 |
| trown|ruck | EmilienM: ^ those are all the ones that have failed on pike | 19:30 |
| trown|ruck | EmilienM: fs17 passed then failed.. so might pass next run | 19:30 |
| trown|ruck | EmilienM: and upload job is probably same as what I am making a bug for on master, but I can check it | 19:30 |
| *** oidgar has quit IRC | 19:30 | |
| tdasilva | mwhahaha: the problem is that I need to get information out of a container and I don't know how that could be done | 19:30 |
| EmilienM | trown|ruck: ok, thanks. I'll let you know if I find something else | 19:32 |
| EmilienM | "qemu-kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory", | 19:33 |
| EmilienM | for the ASK [convert-image : convert image] failure | 19:33 |
| EmilienM | trown|ruck: sounds like something with RDO Cloud maybe | 19:33 |
| EmilienM | on pike it looks like a serious error: | 19:35 |
| EmilienM | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-pike/87b9ed2/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-12-20_12_56_20 | 19:35 |
| *** dsariel has quit IRC | 19:36 | |
| *** jpena is now known as jpena|off | 19:37 | |
| EmilienM | it sounds like a valid puppet error: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-pike/87b9ed2/subnode-2/var/log/journal.txt.gz#_Dec_20_12_56_13 | 19:38 |
| *** rlandy|rover is now known as rlandy|rover|brb | 19:39 | |
| mwhahaha | tdasilva: can you leverage something similar to what we do for the swift rings? | 19:39 |
| EmilienM | this: http://paste.openstack.org/show/629493/ | 19:40 |
| EmilienM | trown|ruck: ^ the puppet error on pike | 19:40 |
| EmilienM | mwhahaha: I'm wondering if we miss a backport here /me digging | 19:40 |
| mwhahaha | let me see | 19:40 |
| EmilienM | it's the rabbitmq bundle thing | 19:40 |
| rook | fultonj: the builtin docker module still consumes tons of memory | 19:41 |
| fultonj | :( | 19:41 |
| mwhahaha | "/usr/bin/docker-current: Error response from daemon: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:258: applying cgroup configuration for process caused \\\\\\\"write /sys/fs/cgroup/pids/system.slice/docker-985b1b2ad4bdb6643087afee7885f72d82fa0a50db03976008df692bec4b2d0d.scope/cgroup.procs: no such device\\\\\\\"\\\"\\n\".", | 19:41 |
| mwhahaha | EmilienM: no there's a bug in docker | 19:41 |
| mwhahaha | i've seen this before, it's not consistent | 19:42 |
| EmilienM | mhh | 19:42 |
| EmilienM | ok maybe but have you seen the rabbitmq thing also? | 19:42 |
| EmilienM | mwhahaha: can you review https://review.openstack.org/#/c/527404/ please? | 19:43 |
| mwhahaha | that's what i'm looking at | 19:43 |
| mwhahaha | in postci | 19:43 |
| tdasilva | mwhahaha: was thinking i could stick a little json file in a swift object somewhat similar to swiftrings | 19:43 |
| mwhahaha | EmilienM: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-pike/87b9ed2/postci.txt.gz | 19:43 |
| tdasilva | mwhahaha: this seems like the perfect job for etcd?? | 19:43 |
| mwhahaha | tdasilva: or stop doing silly stuff as part of the deployment :D | 19:44 |
| trown|ruck | EmilienM: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset008-pike/a991439/undercloud/home/jenkins/failed_deployment_list.log.txt.gz also failed on pike... also looks puppet related | 19:44 |
| tdasilva | mwhahaha: lol, true true | 19:44 |
| mwhahaha | Could not find resource 'Exec[exec-setfacl-manila-manila]' for relationship from 'Ceph::Key[client.manila]' on node upstream-centos-7-2-node-rdo-cloud-tripleo-400-113.localdomain", | 19:45 |
| *** ebarrera has quit IRC | 19:45 | |
| tdasilva | mwhahaha: but this can't be the only case where we are producing dynamic data and setting it to config files, is it? | 19:45 |
| tdasilva | mwhahaha: passwords are generated by mistral, is that correct? | 19:45 |
| mwhahaha | tdasilva: generated prior to deployment | 19:45 |
| mwhahaha | tdasilva: so they are just inputs | 19:46 |
| mwhahaha | tdasilva: and they don't rely on overcloud services | 19:46 |
| tdasilva | yeah, i see, in my case i actually need part of the deployment ready | 19:46 |
| tdasilva | right | 19:46 |
| mwhahaha | tdasilva: right so octavia is the only other instance of something like this really | 19:46 |
| mwhahaha | which is what we're working through and it needs ansible stuff wedged in post deploy | 19:46 |
| mwhahaha | which is really ugly | 19:46 |
| * tdasilva goes to look at octavia | 19:46 | |
| mwhahaha | which is why i said this need to not be a pattern in openstack servers | 19:46 |
| mwhahaha | tdasilva: plz don't, we don't want to repeat that pattern | 19:47 |
| tdasilva | heh | 19:47 |
| *** oidgar has joined #tripleo | 19:47 | |
| mwhahaha | this is where the openstack services need to be able to handle this themselves and not require deployment/config update steps | 19:47 |
| *** oidgar has quit IRC | 19:47 | |
| mwhahaha | swift is awkward here because these is no shared db | 19:48 |
| *** pcaruana has joined #tripleo | 19:48 | |
| tdasilva | mwhahaha: if i wanted to write a script to be executed using bootstrap_host_exec, where would that script live? tripleo-heat-templates? | 19:50 |
| *** jtomasek has quit IRC | 19:50 | |
| mwhahaha | tdasilva: I don't think so because i'm not sure if that's installed by default on the overcloud nodes | 19:51 |
| *** jobewan has joined #tripleo | 19:51 | |
| tdasilva | mwhahaha: ok, let me look at ringbuilder a bit see if I can do something similar | 19:51 |
| mwhahaha | tdasilva: owalsh had to do something simialr and i think (unfortunately) we ended up putting it in tripleo-common or something | 19:53 |
| *** pcaruana has quit IRC | 19:54 | |
| *** moshele has joined #tripleo | 19:55 | |
| Tengu | owalsh: apparently, commenting out the bloc in the overcloud.j2.yaml and deploying did what I needed in order to get back to a stable, working stack. I'll uncomment it right after the deploy is over. | 19:57 |
| * Tengu is happy, because he could correct a really messed up stack | 19:57 | |
| Tengu | btw.... a new check might be interesting in the pre-flight checks. | 19:58 |
| *** dprince has quit IRC | 20:01 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo stable/pike: Correct typo in manila/share.pp resource chaining https://review.openstack.org/529406 | 20:03 |
| EmilienM | mwhahaha: ^ the puppet error that you found - was a missing backport | 20:03 |
| mwhahaha | k | 20:03 |
| * mwhahaha blames gfidente | 20:04 | |
| EmilienM | mwhahaha: we'll need https://review.openstack.org/#/c/527403/ as well (backported) | 20:04 |
| EmilienM | mwhahaha: but https://review.openstack.org/#/c/527404/ first | 20:04 |
| mwhahaha | yea | 20:04 |
| *** oidgar has joined #tripleo | 20:04 | |
| *** oidgar has quit IRC | 20:05 | |
| *** pcaruana has joined #tripleo | 20:06 | |
| *** rmascena__ has joined #tripleo | 20:08 | |
| rook | fultonj: when you get back lemme know. | 20:08 |
| fultonj | what's up rook ? | 20:08 |
| rook | fultonj so, the fsid -- one area I think we can help... the fsid should be the same across nodes. Any reason why we run, and store the fact across all hosts? | 20:09 |
| rook | vs just run on a single node? | 20:09 |
| rook | asking around, I get the sense that fsid is unique per cluster, not per node. | 20:09 |
| openstackgerrit | David Peacock proposed openstack/tripleo-quickstart-extras master: Fix failure of UI validation in some shells https://review.openstack.org/529407 | 20:10 |
| *** rmascena has quit IRC | 20:10 | |
| fultonj | yes it's one per cluster | 20:10 |
| fultonj | how expensive is that operation? | 20:10 |
| fultonj | is it the right thing to optimize? | 20:11 |
| rook | That is the first spike of 39GB (from what I can tell tracing things). | 20:11 |
| *** rmascena__ is now known as raildo | 20:11 | |
| fultonj | wow | 20:11 |
| rook | sorry 37GB* | 20:11 |
| fultonj | big enough | 20:11 |
| rook | moving to docker modules didn't help | 20:11 |
| fultonj | but the docker module was for downloading and starting the image, right? | 20:12 |
| fultonj | pulling it into the local registry | 20:12 |
| rook | right, i was just mentioning that didn't hlep the utilization either. | 20:12 |
| *** pcaruana has quit IRC | 20:13 | |
| openstackgerrit | Matt Young proposed openstack/tripleo-quickstart master: Featureset 22: run tempest (smoke+basic) https://review.openstack.org/529408 | 20:13 |
| *** rlandy|rover|brb is now known as rlandy|rover | 20:14 | |
| fultonj | https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-defaults/tasks/facts.yml#L37 | 20:15 |
| rook | https://github.com/ceph/ceph-ansible/blob/6a9b5c9632a39d290ebf707a21e98f17b064f198/roles/ceph-defaults/tasks/facts.yml#L17 | 20:16 |
| openstackgerrit | Matt Young proposed openstack/tripleo-quickstart master: Featureset 22: run tempest (smoke+basic) https://review.openstack.org/529408 | 20:16 |
| rook | I really wonder if this is the task that causes the problems fultonj ^^ | 20:16 |
| fultonj | seems to record the result, | 20:16 |
| fultonj | or line 52 | 20:16 |
| *** amoralej is now known as amoralej|off | 20:17 | |
| owalsh | mwhahaha, tdasilva: didn't end up in tripleo-common - https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/nova-api.yaml#L126 | 20:17 |
| fultonj | but what would be more resource intensive would be https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-defaults/tasks/facts.yml#L18 | 20:18 |
| mwhahaha | owalsh: oh right bash in the THT | 20:18 |
| fultonj | and then the question would be, can we do without that task | 20:18 |
| owalsh | mwhahaha: probably could add the script to t-h-t and use get_file instead of inline bash script in yaml | 20:19 |
| mwhahaha | owalsh: not sure which is uglier :D | 20:20 |
| rook | fultonj: i think we sent the same thing | 20:20 |
| fultonj | it's name indicates that it checks if ceph is running (gets the fsid as a side effect) | 20:20 |
| rook | fultonj: I do see this in the stdout / mistral log : [WARNING]: scp transfer mechanism failed on [192.168.24.71]. | 20:21 |
| rook | which might be the delegation failing | 20:21 |
| openstackgerrit | Matt Young proposed openstack/tripleo-quickstart master: Featureset 22: run tempest (smoke+basic) https://review.openstack.org/529408 | 20:21 |
| rook | which must not be all that important? | 20:21 |
| owalsh | mwhahaha: yea, reminds me that I meant to move that to tripleo-common when I had more time | 20:21 |
| fultonj | rook: for tripleo... we pass it the fsid | 20:23 |
| fultonj | from heat | 20:23 |
| fultonj | normally ceph-ansible needs to make it and use it but if it's defined... can we skip the task? | 20:23 |
| fultonj | add an extra when | 20:24 |
| fultonj | or line in the when i shoul say | 20:24 |
| rook | fultonj: oh, it is passed??/ | 20:25 |
| fultonj | only for tripleo, but yes | 20:25 |
| rook | so we could add to the when clause. | 20:25 |
| fultonj | i'm not convinced the play's only job is to get the fsid | 20:25 |
| fultonj | but let's try it as a theory | 20:25 |
| openstackgerrit | Mark Hamzy proposed openstack/tripleo-common master: [WIP] Support multiple architectures https://review.openstack.org/528000 | 20:26 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Don't run check-tripleo OVB jobs frm RH1 anymore https://review.openstack.org/526481 | 20:26 |
| rook | fultonj well if it was to really check if ceph is running, failed_when should be true | 20:26 |
| rook | oh, nm... it doesn't have ignore... | 20:27 |
| rook | Maybe have another pre-task to check the state of the container. | 20:27 |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Fix missing attribute in upgrade Infrared plugin https://review.openstack.org/527437 | 20:34 |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Use parameter to control the docker registry env file creation https://review.openstack.org/527438 | 20:35 |
| *** oidgar has joined #tripleo | 20:35 | |
| rook | fultonj: how is the fsid passed? | 20:37 |
| fultonj | rook: ansible-playbook ... --extra-vars {..., "fsid": "2d87a5e8-8e72-11e7-a223-003da9b9b610", ...} | 20:38 |
| fultonj | rook: you can see it in the executor log | 20:38 |
| fultonj | we also pass "generate_fsid": false, | 20:39 |
| fultonj | so that could be the easiest when to add | 20:39 |
| fultonj | assuming that's all that that task does | 20:39 |
| *** oidgar has quit IRC | 20:40 | |
| *** links has joined #tripleo | 20:41 | |
| rook | so the current code is really overwriting the fsid with the same value. | 20:45 |
| *** links has quit IRC | 20:47 | |
| alee | rlandy|rover, EmilienM , mwhahaha, weshay - well this is reassuring -- looks like the rebuild container patches worked (mostly) | 20:49 |
| mwhahaha | worked-ish | 20:49 |
| alee | rlandy|rover, mwhahaha , EmilienM , weshay still confirming that it all got pulled in but ... https://review.openstack.org/#/c/529181/ looks good | 20:50 |
| alee | at least for the zuul check | 20:50 |
| alee | that patch pulls in changes from barbican and nova | 20:51 |
| alee | and all the tests in scenario 2 pass | 20:51 |
| weshay | nice | 20:51 |
| alee | there are some failures in some of the dependent packages -- looking to see what happened | 20:51 |
| alee | mwhahaha, weshay , rlandy|rover where are the logs to show the container rebuilds? | 20:53 |
| *** jcoufal has quit IRC | 20:53 | |
| *** catintheroof has joined #tripleo | 20:54 | |
| weshay | alee, in /homne/zuul overcloud-prep-containers.log | 20:54 |
| weshay | alee, we don't rebuild.. just install the rpm update on the container | 20:54 |
| weshay | rebuilding would take too long | 20:54 |
| *** vpickard is now known as vpickard_ | 20:57 | |
| *** bfournie has quit IRC | 20:58 | |
| *** dprince has joined #tripleo | 21:00 | |
| *** dsariel has joined #tripleo | 21:00 | |
| *** raildo has quit IRC | 21:01 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci master: Update reviewday project list https://review.openstack.org/526712 | 21:02 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: swift_rsync: don't bind mount /run https://review.openstack.org/513020 | 21:03 |
| openstackgerrit | wes hayutin proposed openstack/instack-undercloud master: DNM, undercloud containers TESTING ONLY https://review.openstack.org/518118 | 21:03 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: DNM: Update undercloud install options for containers https://review.openstack.org/517445 | 21:06 |
| alee | weshay, rlandy|rover , mwhahaha , EmilienM so in that last review https://review.openstack.org/#/c/529181/ looking at the overcloud-prep-containers.log, it appears the packages I updated in fact got updated. | 21:09 |
| alee | that is nova-compute and barbican-* | 21:09 |
| weshay | alee, I love it when a plan comes together | 21:09 |
| alee | and all the tests passed | 21:10 |
| rlandy|rover | good news | 21:10 |
| weshay | that's the way to go out in 2017 | 21:10 |
| alee | but, I do not see the logs for those services | 21:10 |
| weshay | mwhahaha, merge it .. merge it | 21:10 |
| weshay | peer opensource pressure | 21:10 |
| alee | in fact the only logs I don't see are the ones that were changed | 21:10 |
| mwhahaha | i think it's already in the gate | 21:11 |
| weshay | oh | 21:11 |
| weshay | merry xmas to everyone then :) | 21:11 |
| mwhahaha | we still need to get packaged tho | 21:11 |
| * mwhahaha doesn't like the pip install in quickstart | 21:11 | |
| weshay | Slower, let's build a rpm together.. at lowes | 21:11 |
| alee | if I am looking in the right place, the logs should be in http://logs.openstack.org/81/529181/1/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/0d670f4/logs/subnode-2/var/log/containers/ ? right? | 21:12 |
| alee | no barbican logs and no nova-compute.log | 21:12 |
| mwhahaha | wonder if the rebuilds break the log mounts | 21:14 |
| openstackgerrit | Mike Fedosin proposed openstack/tripleo-common master: Remove "overcloud-swift-rings" container during overcloud deletion https://review.openstack.org/529414 | 21:14 |
| alee | looks like https://review.openstack.org/#/c/524064/ timed out -- rechecking .. | 21:16 |
| mwhahaha | alee: ah the might explain the missing logs if it timed out before they got collected | 21:17 |
| alee | mwhahaha, no -- thats a different review | 21:17 |
| alee | mwhahaha, I rechecked that one .. that one reported failure | 21:18 |
| *** catinthe_ has joined #tripleo | 21:23 | |
| *** catintheroof has quit IRC | 21:25 | |
| openstackgerrit | Mike Fedosin proposed openstack/tripleo-common master: Remove "overcloud-swift-rings" container during overcloud deletion https://review.openstack.org/529414 | 21:25 |
| alee | mwhahaha, weshay , rlandy|rover there are a number of images that got updated. (not just the ones I modified). It looks like all of those images now lack log files | 21:25 |
| * weshay checks another build | 21:26 | |
| alee | you can see the differences in the same review -- http://logs.openstack.org/36/527136/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/cc76cd0/logs/subnode-2/var/log/containers/ this was from a previous run | 21:26 |
| weshay | alee, honestly this sounds more like a bug w/ containers | 21:27 |
| weshay | alee, we're just updating the rpm | 21:27 |
| weshay | hrm... | 21:28 |
| alee | on the same review. I looked for Running docker command: /usr/bin/docker push" in the overcloud-image-prep.log and searched for the corresponding containers | 21:28 |
| alee | that is - corresponding logs | 21:28 |
| weshay | same thing here | 21:28 |
| weshay | http://logs.openstack.org/72/515372/17/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/98c339f/logs/subnode-2/var/log/containers/nova/ | 21:28 |
| weshay | no compute log | 21:28 |
| weshay | this is problem | 21:28 |
| weshay | will containers ever get yum updated in the field? | 21:28 |
| weshay | alee, we need a lp on this | 21:29 |
| mwhahaha | weshay: this is the thing that needed to get solved for this cycle as how to deploy hotfixes, etc | 21:29 |
| weshay | ya | 21:29 |
| mwhahaha | welcome to containers! | 21:30 |
| alee | weshay, yeah .. you want to file it or shall I? | 21:30 |
| weshay | wooo hooo | 21:30 |
| mwhahaha | were all the old problems are new again | 21:30 |
| weshay | mwhahaha, no worries.. I'm sure another container will fix this | 21:30 |
| mwhahaha | because solving them the first time wasn't a big enough pain in the bit | 21:30 |
| weshay | Slower, get over here | 21:30 |
| weshay | https://www.projectatomic.io/blog/2016/02/dont-run-yum-update-within-a-running-container/ | 21:31 |
| weshay | first hit | 21:31 |
| weshay | although it only says not to because of the time it takes | 21:32 |
| mwhahaha | you wouldn't want to under normal circumstances | 21:33 |
| mwhahaha | because you'd have to do it ever container launch | 21:34 |
| mwhahaha | but it's ok for testing new packages i guess | 21:34 |
| weshay | dmsimard, ping.. when we build the containers in rdo, are we pulling the latest udpates from centos? | 21:34 |
| mwhahaha | you'd want to rebuild | 21:34 |
| weshay | hrm.. is there a thread on the topic that I've missed? | 21:34 |
| dmsimard | weshay: you're pulling from the base centos7 image | 21:34 |
| dmsimard | whatever it is | 21:34 |
| *** catinthe_ has quit IRC | 21:35 | |
| dmsimard | weshay: https://hub.docker.com/r/library/centos/tags/ "7" and "latest" dates from 20 days ago | 21:35 |
| weshay | dmsimard, just qq.. wondering if you have seen this... | 21:35 |
| weshay | say we update openstack/nova on a container.. we loose the nova compute log | 21:35 |
| *** catintheroof has joined #tripleo | 21:35 | |
| weshay | ever see anything like that? | 21:36 |
| *** bnemec has quit IRC | 21:36 | |
| dmsimard | me? I have absolutely no clue, mostly because I haven't worked on the underlying implementation | 21:36 |
| *** threestrands_ has joined #tripleo | 21:36 | |
| weshay | k | 21:36 |
| dmsimard | it might be a question for opstools ? I think they worked on logging in general and on fluentd implementation | 21:36 |
| *** lblanchard has joined #tripleo | 21:37 | |
| dmsimard | mwhahaha, weshay: in a containerized workflow, you usually don't run yum update or apt-get update -- you build a new container and redeploy | 21:37 |
| alee | mwhahaha, got a failure on https://review.openstack.org/#/c/527136/ -- though scenario 2 did succeed .. | 21:37 |
| weshay | dmsimard, yes yes.. this is back to how to do it quickly in ci | 21:38 |
| weshay | dmsimard, you were in on that conversation :) | 21:38 |
| dmsimard | weshay: sure, but you're not supposed to actually run that in a running container | 21:38 |
| Slower | hmm | 21:38 |
| dmsimard | weshay: you need to add a layer which would be like FROM <the image you want to update> RUN yum -y update | 21:38 |
| dmsimard | and then deploy the new layer resulting image | 21:39 |
| alee | mwhahaha, a recheck will prob fix it -- but just confirming .. patches that were based on top of this one succeeded | 21:39 |
| Slower | for the actual yum update that is probably a better idea | 21:39 |
| Slower | I dunno why I did it the way I did now :) | 21:39 |
| dmsimard | OCI is like read only by default | 21:39 |
| dmsimard | so yeah | 21:40 |
| *** catintheroof has quit IRC | 21:40 | |
| Slower | I don't see why it would cause us to lose logs though, that's strange | 21:40 |
| dmsimard | do we have logs of the update ? | 21:40 |
| dmsimard | I have no idea but I'm curious | 21:40 |
| Slower | the only thing I can think is that the metadata on the container changed | 21:41 |
| weshay | http://logs.openstack.org/72/515372/17/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/98c339f/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz | 21:41 |
| weshay | http://logs.openstack.org/72/515372/17/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/98c339f/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz#_2017-12-20_16_40_59 | 21:42 |
| dmsimard | "INFO: Removing container" ? | 21:42 |
| Slower | lots of base OS updates.. | 21:43 |
| Slower | dmsimard: start at container-check | 21:43 |
| *** lblanchard has quit IRC | 21:43 | |
| Slower | hrrm | 21:44 |
| dmsimard | Slower: /usr/bin/docker run --user root --rm 192.168.24.1:8787/tripleomaster/centos-binary-aodh-api:c8cceebf8e648ce46219026f926047491135a66e_fcf8d179 rpm -qa | 21:44 |
| Slower | weshay: we have a few problems here | 21:44 |
| dmsimard | Slower: to me, that reads: "start the aodh-api container if it's not already running, run rpm -qa on it and remove the container once you're done" | 21:45 |
| Slower | dmsimard: so it gets a list of rpms in the container and compares that to the yum database | 21:45 |
| Slower | then updates only containers that need it | 21:45 |
| dmsimard | Slower: was the container already running ? maybe removing it messes up the logging ? I dunno, just brainstorming trying to give ideas | 21:45 |
| Slower | no it wouldn't be running then.. | 21:46 |
| Slower | dmsimard: running it messes up CMD though | 21:46 |
| Slower | /usr/bin/docker run --user root --net host --volume /etc/yum.repos.d:/etc/yum.repos.d --volume /opt:/opt --name yum-update-7 192.168.24.1:8787/tripleomaster/centos-binary-neutron-openvswitch-agent:c8cceebf8e648ce46219026f926047491135a66e_fcf8d179 yum -y update | 21:46 |
| Slower | and then we commit it after with CMD changed to how it should be | 21:47 |
| *** trown|ruck is now known as trown|outtypewww | 21:50 | |
| openstackgerrit | Ian Main proposed openstack/instack-undercloud master: DNM: Testing containerized undercloud. https://review.openstack.org/529419 | 21:53 |
| *** ramishra has quit IRC | 21:59 | |
| *** itlinux_ has joined #tripleo | 22:00 | |
| *** akrivoka has quit IRC | 22:02 | |
| weshay | Slower, dmsimard so any changes needed to container-check? | 22:03 |
| *** Goneri has quit IRC | 22:07 | |
| weshay | alee, mwhahaha the issue is not w/ the update http://logs.openstack.org/25/528125/4/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/ba349fc/logs/subnode-2/var/log/containers/nova/ | 22:08 |
| weshay | I see several reviews w/o nova compute logs | 22:08 |
| weshay | but it did work here... | 22:09 |
| weshay | http://logs.openstack.org/07/472607/159/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/84a4673/logs/subnode-2/var/log/containers/nova/ | 22:09 |
| dmsimard | I WAS TOLD CONTAINERS WOULD SOLVE ALL OF MY PROBLEMS | 22:10 |
| dmsimard | (╯°□°）╯︵ ┻━┻ | 22:10 |
| *** apetrich has quit IRC | 22:11 | |
| *** apetrich has joined #tripleo | 22:11 | |
| weshay | alee, mwhahaha appears to be random http://logs.openstack.org/56/529356/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/5f0dc70/logs/subnode-2/var/log/containers/nova/ | 22:12 |
| alee | weshay, perhaps - but what I saw in this case is that it seems pretty much all the images that were updated did not have logs (at least the ones I checked) | 22:12 |
| weshay | alee, I just pointed out 3 runs w/o updates that did not have logs | 22:13 |
| weshay | yes, it's a problem, no it's not being caused by update | 22:13 |
| alee | weshay, understood -- was there a change that went in over the last couple of days that broke logs? I think we handle the mounts for these in the same way .. | 22:14 |
| weshay | not sure.. going to get a bug going | 22:14 |
| *** fultonj has quit IRC | 22:16 | |
| weshay | https://bugs.launchpad.net/tripleo/+bug/1739492 | 22:17 |
| openstack | Launchpad bug 1739492 in tripleo "nova compute log missing in some containerized deployments" [High,Triaged] | 22:17 |
| *** itlinux_ has quit IRC | 22:18 | |
| *** Goneri has joined #tripleo | 22:23 | |
| *** itlinux_ has joined #tripleo | 22:30 | |
| *** alee is now known as alee_afk | 22:31 | |
| *** rcernin has joined #tripleo | 22:32 | |
| openstackgerrit | waleed mousa proposed openstack/tripleo-heat-templates master: Adding support for role parameters in "environment_generator.py" https://review.openstack.org/529422 | 22:32 |
| *** trozet has quit IRC | 22:32 | |
| *** jappleii__ has joined #tripleo | 22:35 | |
| tbarron | weshay: will containers ever get yum updated in the field? | 22:36 |
| tbarron | weshay: I don't know the plan on this, maybe mburns does? Storage delivers several hotfixes a week sometimes, mostly in cinder. | 22:36 |
| *** threestrands_ has quit IRC | 22:36 | |
| tbarron | weshay: could do docker build and docker push to a registry (where?); then pull from overcloud nodes? | 22:37 |
| tbarron | weshay: the (where?) on the registry is due to the hotfix being customer specific, not a generally published fix. | 22:38 |
| tbarron | weshay: and there are 'test-only' patches, delivered to customers who are willing to try the band-aid and see if it helps | 22:38 |
| *** paramite has quit IRC | 22:41 | |
| dmsimard | tbarron: that's what I was trying to convey earlier | 22:44 |
| tbarron | dmsimard: oh, you probably did then, I was just catching up, reading backlog, and that issue has been on my mind. | 22:45 |
| dmsimard | tbarron: containers are *usually* treated as read only, if you need to do an update, you re-build and re-deploy -- in the worst case scenario, you start from the image you currently have, add a layer (ex: yum update) and then deploy the new image you got from adding that layer | 22:45 |
| tbarron | dmsimard: exactly | 22:45 |
| dmsimard | I don't know what's the intent, but it's usually what people do with containers | 22:45 |
| *** pchavva has quit IRC | 22:46 | |
| dmsimard | building on top of existing layers is proabably the safest route but it can probably lead to bloated images down the road | 22:46 |
| tbarron | dmsimard: I think that's the intent; all mutable content (config, logs, etc. ) are bind mounts form the host | 22:46 |
| tbarron | s/form/from/ | 22:46 |
| tbarron | dmsimard: well, image consolidation is a worthwhile goal but I think it's not an immediate goal | 22:47 |
| tbarron | dmsimard: getting rid of misleading config and packages on the host woule IMO be higher prio | 22:47 |
| tbarron | dmsimard: I'm somewhat concerned that we'll have unanticipated support and maintenance issues | 22:48 |
| dmsimard | tbarron: but anyway, regarding the yum update thing | 22:48 |
| *** jmelvin has quit IRC | 22:48 | |
| tbarron | dmsimard: not an objection to the projectk, but anyways somethihng we'll deal with | 22:48 |
| dmsimard | tbarron: in the context of CI, we might end up in a scenario where we need to rebuild the openstack-nova package because we're doing a depends-on a nova patch for example -- now we'd need to rebuild all the containers with that package in it... but it's actually trickier than that | 22:49 |
| dmsimard | because with some packages (i.e, oslo), you'll find those either in all container images or very early on in the hierarchy tree | 22:50 |
| dmsimard | so you end up having to rebuild all container images which is very expensive from inside a job that is already long | 22:50 |
| tbarron | dmsimard: ack. consider the bugs that are fixed for nova / cinder via the common brick library. | 22:50 |
| tbarron | dmsimard: it's nice that nova can run with its brick and cinder with its brick, but :_ | 22:51 |
| dmsimard | the objective with the yum update workflow in the context of containers in CI is to build the package once, create a local repository and then add a layer which does a yum update on every container image | 22:51 |
| tbarron | :) | 22:51 |
| dmsimard | which is significantly faster and less expensive than a full rebuild | 22:51 |
| *** ManoX has joined #tripleo | 22:51 | |
| dmsimard | now, I don't know the specifics since I'm not intimately involved in that but I was part of the early discussions :) | 22:51 |
| tbarron | dmsimard: got it. Optimizing for time and consistency across containers and then later looking at space/layer consolidation as another pass may make sense. But what do I know? | 22:53 |
| dmsimard | tbarron: I'm not sure if the expectation is to use this kind of workflow in the field | 22:53 |
| dmsimard | tbarron: there's no orchestration around it, it's a dumb yum update.. so there's no notion of sql migrations or whatever | 22:54 |
| *** itlinux_ has quit IRC | 22:54 | |
| *** dsariel has quit IRC | 22:54 | |
| tbarron | dmsimard: well, that's why thinking through the hot-fix and test-fix scenarios needs to be done if it hasn't already been done. | 22:55 |
| tbarron | dmsimard: with the old rpm / build system there was a process for these that was understood, ugly but understood. | 22:55 |
| *** etingof has quit IRC | 22:56 | |
| *** itlinux_ has joined #tripleo | 23:09 | |
| *** dprince has quit IRC | 23:10 | |
| *** etingof has joined #tripleo | 23:11 | |
| *** dhill_ has joined #tripleo | 23:30 | |
| openstackgerrit | Mark Hamzy proposed openstack/tripleo-common master: [WIP] Support multiple architectures https://review.openstack.org/528000 | 23:34 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: Add new roles for Ceph containerization https://review.openstack.org/521989 | 23:38 |
| *** itlinux_ has quit IRC | 23:42 | |
| *** itlinux__ has joined #tripleo | 23:46 | |
| *** rlandy|rover is now known as rlandy|bbl | 23:47 | |
| *** moshele has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!