*** shadower has quit IRC | 00:23 | |
*** sdake has quit IRC | 00:23 | |
*** shadower has joined #tripleo | 00:23 | |
*** shivrao has quit IRC | 00:26 | |
*** tiswanso has joined #tripleo | 00:27 | |
openstackgerrit | Merged openstack/diskimage-builder: Use --nodeps when installing fedora-release https://review.openstack.org/227123 | 00:30 |
---|---|---|
*** rlandy has quit IRC | 00:32 | |
*** sthillma has quit IRC | 00:34 | |
*** tiswanso has quit IRC | 00:46 | |
*** tiswanso has joined #tripleo | 01:00 | |
*** julim has quit IRC | 01:01 | |
*** tiswanso_ has joined #tripleo | 01:02 | |
*** tiswanso has quit IRC | 01:05 | |
*** lazy_prince has joined #tripleo | 01:13 | |
*** julim has joined #tripleo | 01:20 | |
*** killer_prince has joined #tripleo | 01:39 | |
*** lazy_prince has quit IRC | 01:42 | |
*** rhallisey has quit IRC | 01:53 | |
*** julim has quit IRC | 01:58 | |
*** jerrygb has joined #tripleo | 02:09 | |
*** yamahata has quit IRC | 02:26 | |
*** weshay_xchat has quit IRC | 02:29 | |
*** lazy_prince has joined #tripleo | 02:32 | |
*** killer_prince has quit IRC | 02:32 | |
*** shivrao has joined #tripleo | 02:38 | |
openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates: Ensure present/latest for puppet driven package updates https://review.openstack.org/190918 | 02:39 |
openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates: Write package names out to flat files https://review.openstack.org/194348 | 02:39 |
openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates: Make package upgrade pacemaker-aware https://review.openstack.org/225524 | 02:39 |
openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates: yum_update.sh pacemaker and puppet fixes https://review.openstack.org/229710 | 02:39 |
openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates: Force stop a single node pacemaker on yum update https://review.openstack.org/229711 | 02:39 |
openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates: Force stop a single node pacemaker on yum update https://review.openstack.org/229711 | 02:41 |
openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates: Make package upgrade pacemaker-aware https://review.openstack.org/225524 | 02:41 |
*** shivrao has quit IRC | 02:42 | |
*** tzumainn has quit IRC | 02:47 | |
*** lazy_prince has quit IRC | 03:19 | |
*** BharatK has joined #tripleo | 03:21 | |
*** sthillma has joined #tripleo | 03:24 | |
*** tiswanso_ has quit IRC | 03:24 | |
*** Nisha has joined #tripleo | 03:25 | |
*** sthillma_ has joined #tripleo | 03:26 | |
*** sthillma has quit IRC | 03:29 | |
*** sthillma_ is now known as sthillma | 03:29 | |
*** sthillma has quit IRC | 03:40 | |
*** lazy_prince has joined #tripleo | 03:51 | |
*** links has joined #tripleo | 03:55 | |
*** jerrygb has quit IRC | 03:57 | |
*** jerrygb has joined #tripleo | 03:58 | |
*** sdake has joined #tripleo | 04:00 | |
*** Nisha has quit IRC | 04:01 | |
*** sdake_ has joined #tripleo | 04:25 | |
*** aukhan has joined #tripleo | 04:25 | |
*** Nisha has joined #tripleo | 04:25 | |
*** sdake has quit IRC | 04:28 | |
*** jaosorior has joined #tripleo | 04:31 | |
*** masco_ has joined #tripleo | 04:34 | |
*** StevenK has quit IRC | 04:35 | |
*** StevenK has joined #tripleo | 04:36 | |
*** Marga_ has quit IRC | 04:47 | |
*** BharatK has quit IRC | 04:48 | |
*** jaosorior has quit IRC | 05:02 | |
*** BharatK has joined #tripleo | 05:19 | |
*** sdake_ is now known as sdake | 05:21 | |
*** mgrohar has joined #tripleo | 05:40 | |
*** mgrohar has quit IRC | 05:51 | |
*** mgrohar has joined #tripleo | 06:00 | |
*** leanderthal has joined #tripleo | 06:01 | |
*** lsmola has joined #tripleo | 06:06 | |
*** rasca has joined #tripleo | 06:08 | |
*** jaosorior has joined #tripleo | 06:12 | |
*** jaosorior has quit IRC | 06:12 | |
*** jaosorior has joined #tripleo | 06:21 | |
*** shardy has joined #tripleo | 06:25 | |
*** mgrohar has quit IRC | 06:27 | |
*** mgrohar has joined #tripleo | 06:27 | |
*** jprovazn has joined #tripleo | 06:37 | |
*** gfidente has quit IRC | 06:39 | |
*** Marga_ has joined #tripleo | 06:39 | |
*** mgrohar has quit IRC | 06:40 | |
*** matbu_ has joined #tripleo | 06:40 | |
*** matbu has quit IRC | 06:42 | |
*** Marga_ has quit IRC | 06:47 | |
*** olaph has quit IRC | 06:55 | |
*** mcornea has joined #tripleo | 06:56 | |
*** devvesa has joined #tripleo | 06:58 | |
Nisha | pino|work, hello | 07:11 |
Nisha | pino|work, Could you review the patches ? | 07:11 |
*** aufi has joined #tripleo | 07:12 | |
*** akrivoka has joined #tripleo | 07:16 | |
*** penick has joined #tripleo | 07:17 | |
*** penick has quit IRC | 07:24 | |
*** ifarkas has joined #tripleo | 07:25 | |
*** bvandenh has joined #tripleo | 07:36 | |
jprovazn | shardy: morning | 07:37 |
*** jtomasek has joined #tripleo | 07:39 | |
*** jaosorior has quit IRC | 07:42 | |
*** Marga_ has joined #tripleo | 07:44 | |
*** jaosorior has joined #tripleo | 07:45 | |
*** mgrohar has joined #tripleo | 07:45 | |
*** openstackgerrit has quit IRC | 07:46 | |
*** openstackgerrit has joined #tripleo | 07:46 | |
*** Marga_ has quit IRC | 07:48 | |
*** pinotree has joined #tripleo | 07:51 | |
*** pinotree has quit IRC | 07:51 | |
*** pinotree has joined #tripleo | 07:51 | |
*** pinotree has quit IRC | 07:51 | |
*** shivrao has joined #tripleo | 07:52 | |
*** tosky has joined #tripleo | 07:55 | |
*** rlandy has joined #tripleo | 07:56 | |
shardy | jprovazn: hi! | 08:05 |
*** shivrao has quit IRC | 08:05 | |
*** bvandenh has quit IRC | 08:05 | |
*** bvandenh has joined #tripleo | 08:05 | |
pino|work | Nisha: note i still receive notifications via email... also, these reviews have been tiring, a bit more of testing and reading/understanding of people's review could have helped | 08:06 |
*** Nisha has quit IRC | 08:06 | |
*** olaph has joined #tripleo | 08:06 | |
jprovazn | shardy: hi, do you want me to rebase https://review.openstack.org/#/c/227225/ and update the resource name or are you planning to make this change? | 08:08 |
shardy | jprovazn: Hey, I actually already did locally, but I haven't quite finished the subsequent patch yet | 08:08 |
*** derekh has joined #tripleo | 08:08 | |
shardy | jprovazn: I can push what I have, more coming soon | 08:09 |
jprovazn | shardy: ha, awesome! thanks | 08:09 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/python-tripleoclient: Add debug environment to tox https://review.openstack.org/229768 | 08:12 |
openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Wire in NodeExtraConfig interface https://review.openstack.org/227225 | 08:13 |
shardy | jprovazn: ^^ | 08:13 |
* jprovazn looks | 08:13 | |
shardy | jprovazn: I was thinking about zaneb's comment re SoftwareComponent in the subsequent review | 08:13 |
shardy | that could be neater, do you think it's OK to move both registration and unregistration inside the scaled-unit, e.g to the "pre deploy" stage? | 08:14 |
shardy | I think it's probably OK provided it happens after the NetworkDeployment, and I guess it will help if you e.g want a thin image where RPMs are installed by puppet at deployment time from satellite | 08:15 |
shardy | which wouldn't work if registration always happens PostDeploy | 08:15 |
jprovazn | shardy: IIRC my concern was mainly potential issue with network not being set, but if network is already set at that point, I *think* it's fine | 08:16 |
shardy | jprovazn: Ok, I'll wire it in like that and we can see how it goes | 08:16 |
*** jistr has joined #tripleo | 08:18 | |
*** tserong has quit IRC | 08:19 | |
jprovazn | cool | 08:20 |
*** tserong has joined #tripleo | 08:20 | |
*** tserong has joined #tripleo | 08:20 | |
openstackgerrit | Pino Toscano proposed openstack/python-tripleoclient: image build: add support for virt-dib https://review.openstack.org/222541 | 08:27 |
openstackgerrit | Yanis Guenane proposed openstack/tripleo-puppet-elements: Add puppet-logrotate and puppet-rsyslog to the list of modules https://review.openstack.org/227866 | 08:36 |
*** paramite has joined #tripleo | 08:37 | |
*** killer_prince has joined #tripleo | 08:37 | |
openstackgerrit | Derek Higgins proposed openstack/tripleo-common: tripleo.sh Add option to locally build delorean package https://review.openstack.org/228157 | 08:38 |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Use tripleo.sh to build packages https://review.openstack.org/229425 | 08:38 |
*** lazy_prince has quit IRC | 08:41 | |
*** Marga_ has joined #tripleo | 08:45 | |
*** Marga_ has quit IRC | 08:50 | |
*** BharatK has quit IRC | 08:51 | |
*** killer_prince has quit IRC | 08:56 | |
*** lazy_prince has joined #tripleo | 08:56 | |
openstackgerrit | Merged openstack/tripleo-common: Remove support for Tuskar https://review.openstack.org/223527 | 08:56 |
*** gfidente has joined #tripleo | 08:57 | |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Use tripleo.sh to build packages https://review.openstack.org/229425 | 08:58 |
*** jaosorior has quit IRC | 08:58 | |
openstackgerrit | Pino Toscano proposed openstack/python-tripleoclient: image build: add --builder-extra-args https://review.openstack.org/229778 | 09:00 |
*** BharatK has joined #tripleo | 09:04 | |
openstackgerrit | Merged openstack/python-tripleoclient: Actually call _validate_args to check for network types mismatch https://review.openstack.org/221321 | 09:05 |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add flake8 to requirements https://review.openstack.org/229780 | 09:10 |
derekh | shardy: sorry, this is going to be a vague question | 09:11 |
shardy | lol | 09:12 |
derekh | shardy: there was a issue mentioned a while back when stack deployments were not continuing, as if heat didn't reciever a signal for one of the steps | 09:12 |
derekh | shardy: that that ring any bells ? | 09:13 |
*** shardy_ has joined #tripleo | 09:13 | |
shardy_ | derekh: Yeah, there were some internal issues with software deployment signalling (race condition) | 09:13 |
shardy_ | derekh: AFAIK they are all fixed now, let me get the bug, sec | 09:13 |
derekh | shardy_: cool, ya thats what I'm looking for, thanks | 09:14 |
*** shardy has quit IRC | 09:14 | |
derekh | shardy_: I've gotten fixes for the HA job, that gets it back to somtimes passing | 09:15 |
derekh | shardy_: the times it doesn't pass, heat returns a http 400 to one of the signals during a step and everything stops | 09:16 |
derekh | shardy_: so I'm just wondering if we have those fixes in the version of heat we are using | 09:16 |
*** shardy_ has quit IRC | 09:18 | |
*** shardy has joined #tripleo | 09:19 | |
*** Nisha has joined #tripleo | 09:21 | |
openstackgerrit | Jan Provaznik proposed openstack/tripleo-common: Fix wrong node count when deleting specific nodes https://review.openstack.org/226682 | 09:21 |
*** mgrohar has quit IRC | 09:22 | |
*** andrearosa has joined #tripleo | 09:23 | |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Test a update in trunk repository version https://review.openstack.org/229789 | 09:24 |
openstackgerrit | Jan Provaznik proposed openstack/tripleo-specs: Spec for running overcloud nodes reboot https://review.openstack.org/224481 | 09:25 |
pino|work | d0ugal: hi, do you know what's going on in https://review.openstack.org/#/c/229778/ wrt py34? | 09:28 |
openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move RHEL (un)registration to NodeExtraConfig https://review.openstack.org/227226 | 09:29 |
shardy | jprovazn: ^^ That just moves the registration/unregistration to NodeExtraConfig | 09:29 |
shardy | I'll do a follow-up patch combining the configs into a SoftwareComponent | 09:29 |
d0ugal | pino|work: Not sure, I seen that before on another review and it only happened once | 09:29 |
jprovazn | shardy: cool, thanks, /me tries | 09:29 |
d0ugal | pino|work: so hopefully a recheck will solve it :) | 09:29 |
pino|work | d0ugal: your recheck failed the same way :/ | 09:30 |
shardy | jprovazn: I can't test the actual registration, but I stubbed the scripts with an echo/exit and that works OK | 09:30 |
d0ugal | pino|work: Really< it doesn't look like it has finished to me | 09:30 |
d0ugal | pino|work: or I can't see the result | 09:30 |
d0ugal | pino|work: but in that case, I don't know. sorry | 09:30 |
*** andrearosa has quit IRC | 09:30 | |
pino|work | d0ugal: https://jenkins03.openstack.org/job/gate-python-tripleoclient-python34/4/ | 09:30 |
pino|work | oh np, just wondering whether was something known | 09:30 |
jprovazn | shardy: ah, great, good to know | 09:31 |
jprovazn | mtg, bbl | 09:31 |
*** jtomasek has quit IRC | 09:35 | |
*** bvandenh has quit IRC | 09:37 | |
*** sdake_ has joined #tripleo | 09:38 | |
*** pelix has joined #tripleo | 09:41 | |
shardy | Does anyone want to review https://review.openstack.org/#/q/status:open+project:openstack/tripleo-heat-templates+branch:master+topic:bug/1463092,n,z before it lands? | 09:41 |
*** sdake has quit IRC | 09:42 | |
shardy | there are three patches (puppet & update related) that look ready to land and are passing CI | 09:42 |
* shardy will approve in a bit if nobody objects | 09:42 | |
*** Marga_ has joined #tripleo | 09:46 | |
openstackgerrit | Derek Higgins proposed openstack/tripleo-common: tripleo.sh Add option to locally build delorean package https://review.openstack.org/228157 | 09:46 |
*** bvandenh has joined #tripleo | 09:49 | |
*** Marga_ has quit IRC | 09:51 | |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 09:51 |
*** jtomasek has joined #tripleo | 09:51 | |
*** jaosorior has joined #tripleo | 09:52 | |
*** mcornea has quit IRC | 09:53 | |
*** jaosorior has quit IRC | 09:54 | |
*** jaosorior has joined #tripleo | 09:54 | |
*** jaosorior has quit IRC | 09:54 | |
*** jaosorior has joined #tripleo | 09:55 | |
*** jtomasek has quit IRC | 09:57 | |
*** mcornea has joined #tripleo | 10:06 | |
spredzy | I am trying to run tripleo.sh, but it fails when doing so os-net-config setup. It is looking for ethX when nics are named enpXsY. Anyone already had that ? | 10:16 |
spredzy | shardy, ^ | 10:16 |
shardy | spredzy: are you installing the undercloud on baremetal? | 10:17 |
spredzy | the tripleo.sh is ran on a baremetal, I ran it with --all option | 10:17 |
spredzy | shardy, | 10:17 |
shardy | spredzy: I think for baremetal you need to configure the undercloud.conf local_interface | 10:19 |
shardy | so you may need to run each step (not --all) and do the undercloud install manually | 10:19 |
shardy | IIRC the undercloud defaults work for the instack-virt-setup VMs, but we don't attempt to detect baremetal nic names | 10:19 |
shardy | spredzy: also FYI last time I tried the introspection wasn't working, that may be fixed now tho | 10:20 |
spredzy | shardy, ack, will run the step one by one. Thanks for the info | 10:21 |
shardy | spredzy: we should probably add an UNDERCLOUD_LOCAL_INTERFACE variable to the script I guess | 10:21 |
openstackgerrit | Merged openstack/tripleo-heat-templates: Write package names out to flat files https://review.openstack.org/194348 | 10:22 |
spredzy | shardy, IMO one should be able to get the script run it and it should work out of the box with the default values, then if neede tweak it at which | 10:22 |
spredzy | wish* | 10:22 |
*** hashar has joined #tripleo | 10:23 | |
shardy | spredzy: well for VMs it does, but it's probably quite hard to automatically detect which nic to use on baremetal | 10:32 |
shardy | e.g if you choose the wrong one you can hose the local network w/dhcp, or take down the primary nic connecting to the local network, etc | 10:33 |
shardy | spredzy: maybe we should detect that we don't know and prompt the user tho | 10:33 |
shardy | Or, at least fail when the default nic doesn't even exist ;) | 10:33 |
shardy | https://review.openstack.org/#/c/225096/8/scripts/tripleo.sh | 10:33 |
shardy | I made similar comments here, FWIW | 10:34 |
*** paramite is now known as paramite|afk | 10:35 | |
*** andrearosa has joined #tripleo | 10:37 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/python-tripleoclient: Remove unused cli arguments from overcloud deploy https://review.openstack.org/229815 | 10:37 |
*** andrearosa has left #tripleo | 10:38 | |
*** paramite|afk is now known as paramite | 10:39 | |
*** Nisha_away has joined #tripleo | 10:40 | |
spredzy | shardy, agree, if one has a box for different purpose one of them being tripleo, it might be a bad idea to auto-guess the NIC | 10:40 |
*** Nisha has quit IRC | 10:40 | |
*** BharatK has quit IRC | 10:42 | |
openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates: Keystone domain for Heat https://review.openstack.org/180566 | 10:44 |
*** Nisha_away has quit IRC | 10:44 | |
*** Nisha_away has joined #tripleo | 10:45 | |
*** Marga_ has joined #tripleo | 10:46 | |
*** Marga_ has quit IRC | 10:51 | |
openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-heat-templates: Make puppet manifests compliant with Puppet 4.x https://review.openstack.org/225664 | 10:52 |
*** jaosorior has quit IRC | 10:54 | |
*** BharatK has joined #tripleo | 10:55 | |
*** Nisha_away has quit IRC | 10:57 | |
*** trown|outttypeww is now known as trown | 11:00 | |
*** paramite is now known as paramite|afk | 11:02 | |
*** aukhan has quit IRC | 11:07 | |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Test a update in trunk repository version https://review.openstack.org/229789 | 11:12 |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Remove sudo when calling tripleo.sh https://review.openstack.org/229823 | 11:12 |
derekh | Hmm, the repository in delorean-deps seems to have vanished, causing CI falures http://cbs.centos.org/repos/cloud7-openstack-common-testing/x86_64/os/Packages/ | 11:21 |
trown | ruh roh... that happened over the weekend too | 11:24 |
*** paramite|afk is now known as paramite | 11:29 | |
slagle | oh geez | 11:32 |
slagle | i'm eating my words when i was arguing for the reliability of the rdo infra | 11:33 |
derekh | slagle: its not rdo infra, its cenots infra | 11:34 |
slagle | oh right, thank goodness | 11:34 |
derekh | slagle: ;-) | 11:34 |
derekh | slagle: looks like when they generate a new repo (the one specified in delorean-deps), its not an atomic operation | 11:35 |
derekh | slagle: so while it happens there is now repo .... | 11:35 |
slagle | i'd like to find a way to generate the commands in the docs from the tripleo.sh code | 11:38 |
slagle | was thinking about using the same awk stuff we have in incubator | 11:39 |
derekh | slagle: easy $ cat tripleo.sh | 11:39 |
slagle | except i dont want all the comments and actual documentation to live in tripleo.sh | 11:39 |
slagle | so i was thinking a way to mark just the line(s) in tripleo.sh that "install the undercloud", then parse that out somehow, and add the result into the tripleo-docs rst | 11:40 |
derekh | slagle: some kind of tagged markers | 11:41 |
slagle | right, something like that | 11:41 |
derekh | slagle: sounds good | 11:42 |
slagle | i'm going to add this bit to the trello card | 11:45 |
slagle | this feels like concensus! | 11:45 |
*** dprince has joined #tripleo | 11:47 | |
*** dtantsur|afk is now known as dtantsur | 11:47 | |
*** Marga_ has joined #tripleo | 11:47 | |
*** Nisha has joined #tripleo | 11:50 | |
*** jtomasek has joined #tripleo | 11:51 | |
*** Marga_ has quit IRC | 11:52 | |
openstackgerrit | Merged openstack/instack-undercloud: Add basic API docs https://review.openstack.org/223810 | 11:57 |
derekh | Ok, so the HA job has 3 problems I know of the fix for the first is here https://review.openstack.org/#/c/229183/ once thats merged I'll track down the 2nd | 12:00 |
derekh | which I believe to be something todo with https://github.com/puppetlabs/puppetlabs-mysql/commit/97b8200a5f93d1a6853a8994229183706e6e989e | 12:00 |
derekh | With those two fixed the HA job sometimes passes again (e.g. here https://jenkins05.openstack.org/job/gate-tripleo-ironic-overcloud-f21puppet-ha/174/) | 12:00 |
derekh | I things the answer to the 3rd problem causing it to sometimes fail is somewhere in heat or our signaling mechanism | 12:01 |
trown | I had noticed it passed a couple times, and was wondering how | 12:02 |
*** rhallisey has joined #tripleo | 12:03 | |
*** rbrady has joined #tripleo | 12:04 | |
*** marcusvrn_ has joined #tripleo | 12:06 | |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 12:07 |
pino|work | "nothing to see here" is the best way of saying "everybody, look HERE!" :) | 12:08 |
openstackgerrit | Merged openstack/os-cloud-config: Fix nodes registering with fake_pxe driver https://review.openstack.org/223046 | 12:08 |
openstackgerrit | James Slagle proposed openstack/python-tripleoclient: Make ssh user configurable https://review.openstack.org/228413 | 12:10 |
openstackgerrit | Merged openstack/tripleo-heat-templates: Ensure present/latest for puppet driven package updates https://review.openstack.org/190918 | 12:10 |
*** pblaho has joined #tripleo | 12:11 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: Make package upgrade pacemaker-aware https://review.openstack.org/225524 | 12:11 |
*** jtomasek has quit IRC | 12:11 | |
*** morazi has joined #tripleo | 12:12 | |
*** tzumainn has joined #tripleo | 12:16 | |
*** masco_ has quit IRC | 12:17 | |
*** jerrygb has quit IRC | 12:17 | |
*** jtomasek has joined #tripleo | 12:17 | |
*** jaosorior has joined #tripleo | 12:18 | |
*** jayg|g0n3 is now known as jayg | 12:20 | |
openstackgerrit | Jan Provaznik proposed openstack/python-tripleoclient: Change RHEL subscription template path https://review.openstack.org/229847 | 12:24 |
*** weshay_xchat has joined #tripleo | 12:25 | |
*** weshay_xchat is now known as weshay | 12:27 | |
*** pradk has joined #tripleo | 12:28 | |
*** spzala has joined #tripleo | 12:31 | |
openstackgerrit | Merged openstack/python-tripleoclient: Fix file checksum function in for python 3.4 https://review.openstack.org/228608 | 12:32 |
*** mkovacik has joined #tripleo | 12:32 | |
*** BharatK has quit IRC | 12:35 | |
shardy | zaneb: Hey, I pushed updates with the NodeExtraConfig rename we discussed yesterday | 12:45 |
shardy | zaneb: I've not forgotten your comments re SoftwareComponent in https://review.openstack.org/#/c/227226/ | 12:45 |
shardy | I just decided it might be cleaner to handle that refactor in a subsequent patch | 12:45 |
zaneb | shardy: np, I think it's much easier to move that later than to rename stuff later | 12:48 |
*** Marga_ has joined #tripleo | 12:48 | |
shardy | zaneb: +1, I'll ping the ML about the more general rename later, e.g potentially removing the *Pre suffix | 12:48 |
*** BharatK has joined #tripleo | 12:49 | |
shardy | zaneb: we can probably deprecate them and leave some aliases to the old names in place, but it's unfortunate that there's no way to generate a user-visible warning | 12:49 |
shardy | maybe we could add something to the oscplugin which warns for a couple of cycles, then remove the aliases | 12:49 |
zaneb | shardy: I actually don't think living with the ones we've got is a big problem | 12:50 |
zaneb | I was more worried about minimising any confusion with new ones at a stage where we still had time to fix them | 12:51 |
shardy | zaneb: ack, I guess I was looking more for consistency, but perhaps we can have both in due course | 12:51 |
shardy | zaneb: I'll work up a docs patch to at least explain what we have, as it's not all captured atm | 12:51 |
zaneb | +1 great idea | 12:51 |
* zaneb was certainly blundering around cluelessly | 12:52 | |
*** julim has joined #tripleo | 12:52 | |
*** mgrohar has joined #tripleo | 12:53 | |
*** Marga_ has quit IRC | 12:53 | |
dprince | slagle, shardy: commented here https://review.openstack.org/#/c/222772/5 | 12:58 |
dprince | couple of thoughts on this (interesting) topic. First is would it make sense to put the 'fake server' resource into Heat itself somehow... | 12:59 |
dprince | because a pre-existing server resource in Heat would expose new options for using Heat for configuring only | 13:00 |
shardy | dprince: probably not, because if it's an actual nova resource, we should do something like https://review.openstack.org/#/c/135492/ instead | 13:00 |
shardy | the dummy server thing is really only needed when the server is something not known to openstack at all | 13:00 |
shardy | dprince: I think the external_id interface proposed by asalkeld pretty much does what you just described | 13:01 |
dprince | shardy: I don't know if I care about it being a Nova resource | 13:01 |
dprince | shardy: cool, this would be perhaps valuable if we go down this route | 13:02 |
shardy | dprince: In that case, heat doesn't have to care about it, you can just use an empty nested stack, as I did in the examples I provided for slagle | 13:02 |
dprince | shardy: yep, got it | 13:02 |
shardy | https://etherpad.openstack.org/p/noop-softwareconfig | 13:02 |
dprince | the second idea is what if we architected our templates differently so that we always did this | 13:02 |
dprince | shardy: ^^ | 13:03 |
shardy | dprince: Yeah, it's certainly worth thinking about, and it makes perfect sense for some things that can be independently created, e.g Ceph clusters | 13:03 |
shardy | I'm less sure about having a series of separate stacks operating on the same servers, because you lose the dependency information Heat normally has between nested stacks | 13:04 |
shardy | could be worth investigating though - I'm definitely +1 on anything which gets us incrementally away from the giant stack architecture | 13:04 |
dprince | shardy: here is the thing I actually like about that... it gives us options for refactoring in t-h-t | 13:04 |
dprince | shardy: our puppet manifests are mostly idempotent, or they should be | 13:05 |
dprince | shardy: so it doesn't really matter if I pull the rug on the the Heat config resources and re-run them | 13:05 |
dprince | shardy: if you buy that... then I think there could be serious merit to always managing our stacks this way | 13:06 |
shardy | dprince: I dunno, I guess I see it as decoupling something which is in reality still coupled, you'd need something orchestrating each heat stack (wrapper script) which would in the end do exactly what the parent stack already does | 13:07 |
*** Goneri has joined #tripleo | 13:07 | |
shardy | I think I'd need to see an example to form a proper opinion, but like I said, probably worth investigating :) | 13:07 |
slagle | dprince: i actually started hacking on a fake server resource in heat initially, until shardy showed me how easy this was without that | 13:09 |
dprince | shardy: ack, I may try this dual stack idea out | 13:09 |
slagle | but my goal here was to make it work on a server not known to nova. that being said, i think it could be adapted to work both ways. servers known to nova, but not orchestrated via heat, and servers not known to nova at all | 13:09 |
dprince | slagle: cool, just asking the question | 13:09 |
slagle | sure. i'm actually super pleased with how well this is working | 13:10 |
dprince | slagle: cool. The idea I mentioned about was to use 2 stacks: one to create the resources, and another to configure them | 13:11 |
dprince | slagle: in the second stack (the configuration stack) we would essentially always use your new code | 13:11 |
slagle | yea, makes sense | 13:11 |
*** athomas has quit IRC | 13:11 | |
dprince | slagle: would be worth prototyping | 13:11 |
dprince | slagle: how it effects things like network isolation, etc would be potentially trick. But I actually think doing this would help the Spine Leaf architecture concerns too | 13:12 |
dprince | slagle: because the single stack... isn't going to handle that so well I'm afraid | 13:12 |
slagle | just b/c we don't always have a full provisioning network to all the nodes in spine/leaf? | 13:13 |
dprince | spine leaf is like network isolation for each rack | 13:13 |
dprince | parameter explosion!!! (or obfuscation) if we do it in a single stack perhaps | 13:13 |
dprince | shardy: I also think the dual stack approach could have merit for other reasons as well | 13:15 |
*** athomas has joined #tripleo | 13:17 | |
*** lazy_prince has quit IRC | 13:18 | |
*** noslzzp has joined #tripleo | 13:18 | |
jayg | jistr: ping, qq - is the NeutronScale resource not included in tht on purpose? if so, do you know the reason? it is still in the ref arch is why I ask | 13:19 |
*** rpothier has joined #tripleo | 13:23 | |
slagle | dprince: what did we end up deciding about requiring adding parameters at the top level? EmilienM has this patch: https://review.openstack.org/229471 and was asking if he needed to update overcloud-without-mergey.yaml as well. i don't recall what the decision here was | 13:24 |
dprince | slagle: I'm torn on those being top level | 13:25 |
*** Nisha has quit IRC | 13:25 | |
*** mcornea has quit IRC | 13:25 | |
shardy | slagle: FWIW my understanding was that new parameters to the role templates, e.g puppet/controller.yaml should be surfaced via overcloud-without-mergepy, and the parameter_defaults should be used only for optional additions, e.g *ExtraConfig | 13:25 |
shardy | slagle: Or, maybe that's just what I was thinking in my head ;) | 13:25 |
dprince | shardy: sort of, but if we do that we are going to get a slew of obscure top level parameters | 13:26 |
shardy | IMHO it will get very confusing if we encourage paramter-defaults to be wired in to the "core" templates, vs for third-party additions | 13:26 |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Ensure access to RabbitMQ is password protected https://review.openstack.org/210402 | 13:26 |
dprince | if the paramter is in any sense implementation specific (related to puppet or docker only) they don't belong at the top level | 13:27 |
shardy | dprince: In the patch slagle referenced, couldn't those just be added to the static hieradata yaml? | 13:27 |
dprince | shardy: yes. I think so | 13:27 |
dprince | shardy: although that is arguably a bit of an advanced feature | 13:27 |
shardy | dprince: Well, it can be overriden easily enough via the controllerExtraConfig parameter? | 13:28 |
dprince | shardy: My only concern is having too many top level parameters | 13:28 |
shardy | dprince: one option would be to start making use of heat ParameterGroups | 13:28 |
dprince | shardy: okay, perhaps you can ask that question on the review | 13:28 |
shardy | then parameters could be grouped and categorized as "core" and "advanced", or whatever | 13:29 |
dprince | shardy: I'd be fine using that for these sort of configurations... probably just need to document it somewhere in the "puppet specific" docs | 13:29 |
shardy | dprince: Yeah, I'm planning a docs patch which refreshes the various ExtraConfig (parameter and resource) interfaces | 13:29 |
dprince | shardy: yeah, shadower used to have a TODO in the templates to start using parameter groups at one point | 13:29 |
slagle | shardy: i think i'm with you and i do remember agreeing that the role template parameters should be surfaced. that being said, we can always revisit | 13:29 |
shardy | what we have atm is a bit outdated now | 13:29 |
*** tiswanso has joined #tripleo | 13:29 | |
*** jprovazn has quit IRC | 13:29 | |
*** bvandenh has quit IRC | 13:29 | |
*** lblanchard has joined #tripleo | 13:31 | |
jistr | jayg: i recall there were some issues with neutron service names and how they reported themselves to neutron API... could that be related? | 13:32 |
jistr | i don't recall exactly what the problem was though | 13:32 |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Parametrize RabbitMQ FD limit https://review.openstack.org/201796 | 13:32 |
dprince | shardy: ++ for Parameter groups at the top level | 13:33 |
jistr | jayg: so i think it's quite possible that NeutronScale is not there on purpose | 13:33 |
jistr | jayg: yup here is the removal of NeutronScale https://github.com/openstack/tripleo-heat-templates/commit/1ca7efc5d18c4b5ba5f53753dcfbbe790c42aaad | 13:35 |
*** trown is now known as trown|brb | 13:35 | |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Ensure access to RabbitMQ is password protected https://review.openstack.org/210402 | 13:36 |
gfidente | jayg, I think refarch was meant to support rhos6 where metadata was supposed to take over the hostname from the other hosts | 13:37 |
gfidente | jayg, with rhos7 we didn't need to that | 13:37 |
*** egafford has joined #tripleo | 13:39 | |
d0ugal | slagle: For tripleo.sh, is the usage just clone the repo and run on the undercloud? or? | 13:40 |
derekh | d0ugal: http://paste.openstack.org/show/475071/ | 13:41 |
d0ugal | akrivoka: ^ | 13:41 |
d0ugal | derekh: Thanks | 13:42 |
mgrohar | hey | 13:42 |
akrivoka | thanks | 13:42 |
mgrohar | can anyone what is the minumum number of controllers for openstack HA in rhos7 | 13:42 |
mgrohar | is it 3x? | 13:43 |
akrivoka | derekh: so in case of virt deployment, things like register nodes and overcloud deploy are also run from the virt host and not from the undercloud vm? | 13:43 |
slagle | d0ugal: yes. i plan on getting this documented at some point | 13:43 |
mgrohar | or do tripleo-heat-templates support 2x controllea HA in active-passive way? | 13:43 |
d0ugal | slagle: Thanks | 13:44 |
slagle | d0ugal: but that's the idea anyway. it'll be for stuff that doesnt have to run on the undercloud as well, eventually | 13:44 |
slagle | such as the virt setup, and shardy has patches up for the delorean package building too | 13:44 |
d0ugal | Neat | 13:44 |
derekh | akrivoka: I've been creating an undercloud vm with instack-virt-setup, then ssh to undercloud and run everything else from there | 13:46 |
akrivoka | derekh: I see, thanks | 13:47 |
jayg | jistr gfidente: ok, thanks, do you happen to know if this was discussed with fabio and andrew (and anyone from neutron team, since they added the scale service)? if it is truly not needed anymore, then maybe we can get the ref arch updated to match reality | 13:47 |
gfidente | with amuller and fabbione yes | 13:47 |
gfidente | and marios actually :) | 13:47 |
*** trown|brb is now known as trown | 13:47 | |
*** Marga_ has joined #tripleo | 13:49 | |
jayg | gfidente: ok, cool, then maybe I'll check in with fabbione (he can notice here if around) to see if he thinks that NeutronScale is needed in the refarch docs anymore based on what you have already discussed | 13:51 |
gfidente | jayg, yeah I am unsure what will happen with the refarch | 13:52 |
gfidente | given I think there is desire to shift over a different process | 13:53 |
gfidente | where bugs are filed as required when there is something to be changed | 13:53 |
*** Marga_ has quit IRC | 13:53 | |
*** jprovazn has joined #tripleo | 13:54 | |
openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Deprecate the undercloud templates. https://review.openstack.org/229893 | 13:54 |
*** links has quit IRC | 13:57 | |
jayg | gfidente: ok, that seems reasonable, I just noticed the diff when comparing tht to refarch for a sprint task I am working on, wanted to see what the reasoning was | 14:03 |
dprince | shardy: are you good here now https://review.openstack.org/#/c/226981/ | 14:09 |
dprince | shardy: if you have a moment you might also review this one https://review.openstack.org/#/c/226968/ | 14:10 |
*** david-lyle has quit IRC | 14:11 | |
shardy | dprince: minor issue with the default in 226981 otherwise lgtm | 14:12 |
dprince | shardy: is an empty list not valid then? | 14:13 |
*** david-lyle has joined #tripleo | 14:13 | |
shardy | dprince: I assumed it wasn't, because the value parsed by that type is a comma delimited string | 14:13 |
shardy | let me check, sec | 14:14 |
dprince | shardy: hmmm, it is valid Json. SO I would assume also a valid list... | 14:14 |
shardy | dprince: comma_delimited_list accepts a string, json accepts either a json string, or a yaml map/list | 14:14 |
dprince | shardy: I think this is why I liked json better :) | 14:17 |
openstackgerrit | Derek Higgins proposed openstack/tripleo-common: Fix repo whitelist for heat and ironic https://review.openstack.org/229906 | 14:18 |
derekh | So we havn't been pulling heat and ironic from the repo we thougt we were pulling it from ^ | 14:19 |
shardy | dprince: Hmm, turns out I'm wrong, it actually works fine, apologies! | 14:19 |
dprince | shardy: I was gonna say, I think I tested this... | 14:19 |
openstackgerrit | Merged openstack/os-net-config: ifcfg: Add support for dns_servers https://review.openstack.org/226968 | 14:20 |
shardy | https://github.com/openstack/heat/blob/master/heat/engine/parameters.py#L359 | 14:21 |
*** mgrohar has quit IRC | 14:21 | |
shardy | dprince: yeah, we actually look for a list, then fall back to splitting a string | 14:21 |
shardy | which is more permissive than I remembered ;) | 14:21 |
spredzy | jistr, when running openstack overcloud image upload I have an error stating ironic-python-agent.initramfs does not exist, does that ring a bell to you ? | 14:23 |
jistr | spredzy: yes | 14:23 |
jistr | mmt | 14:23 |
jistr | spredzy: https://github.com/rdo-management/inlunch/blob/62bd434381392814450e201203d77b4baf802ac1/answers.yml.example#L124 | 14:24 |
jistr | spredzy: you could also try the new tripleo.sh script i think. paramite have you had success with it in the end? | 14:25 |
jistr | spredzy: tripleo.sh would do that automatically | 14:25 |
*** leanderthal has quit IRC | 14:27 | |
spredzy | jistr, thanks for the fix, I ran tripleo.sh this morning, but couldn't get what I needed out of it, need to restart it later today on the box | 14:27 |
openstackgerrit | Ryan Brady proposed openstack/tripleo-common: [WIP]Adds initial swift integration for plans https://review.openstack.org/227297 | 14:28 |
*** paramite is now known as paramite|afk | 14:34 | |
dprince | marios: are you also okay with this (similar to the instack change): https://review.openstack.org/#/c/226327/ | 14:36 |
dprince | slagle: you are still cool here right? https://review.openstack.org/#/c/226981/ | 14:36 |
marios | dprince: looking | 14:37 |
openstackgerrit | Merged openstack/puppet-tripleo: Automatically install packages when upgrading https://review.openstack.org/228532 | 14:38 |
openstackgerrit | Merged openstack/instack-undercloud: swift::storage::all deprecation cleanup https://review.openstack.org/226315 | 14:38 |
marios | dprince: not sure if you wana wait for ci on https://review.openstack.org/#/c/226327/1 but last run fail is not relevant afaik | 14:41 |
openstackgerrit | Ethan Gafford proposed openstack/tripleo-heat-templates: Sahara Integration https://review.openstack.org/220863 | 14:41 |
marios | dprince: so i leave the recheck up to you | 14:41 |
dprince | marios: cool. thanks | 14:42 |
dprince | slagle: as I'm going through reviews I'm abandoning (and marking tickets as won't fix) for anything older than 6 months | 14:45 |
dprince | there aren't that many reviews to just do this manually | 14:45 |
openstackgerrit | Merged openstack/os-cloud-config: register-nodes: add support for pxe_ucs https://review.openstack.org/218939 | 14:45 |
openstackgerrit | Merged openstack/tripleo-heat-templates: swift::storage::all deprecation cleanup https://review.openstack.org/226327 | 14:46 |
*** Marga_ has joined #tripleo | 14:50 | |
*** jtomasek has quit IRC | 14:50 | |
slagle | dprince: wfm. | 14:51 |
spredzy | jistr, bothering you again. Did you encouter this issue while running bulk start ? OperationalError: (OperationalError) no such table: nodes u'DELETE FROM nodes WHERE nodes.uuid = ?' (...74236a566ad3',) | 14:51 |
jistr | spredzy: yes :) | 14:51 |
* spredzy smiled | 14:52 | |
dprince | slagle: this again too https://review.openstack.org/#/c/226981/ | 14:52 |
*** Marga_ has quit IRC | 14:54 | |
openstackgerrit | Merged openstack/tripleo-specs: Spec for TripleO overcloud deployment library https://review.openstack.org/219754 | 14:58 |
openstackgerrit | Merged openstack/tripleo-heat-templates: Add DnsServers param to network config templates https://review.openstack.org/226981 | 14:59 |
*** aufi has quit IRC | 14:59 | |
*** untriaged-bot has joined #tripleo | 15:00 | |
untriaged-bot | Untriaged bugs so far: | 15:00 |
untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1499298 | 15:00 |
openstack | Launchpad bug 1499298 in tripleo "Installers should use pymysql db driver as of Liberty" [Undecided,New] | 15:00 |
untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1501378 | 15:00 |
openstack | Launchpad bug 1501378 in tripleo "pacemaker neutron resource constraints and cleanup" [Undecided,Triaged] | 15:00 |
untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1501296 | 15:00 |
openstack | Launchpad bug 1501296 in tripleo "Stack updates fail due to a new resource type" [Medium,New] | 15:00 |
*** untriaged-bot has quit IRC | 15:00 | |
*** hashar has quit IRC | 15:00 | |
*** paramite|afk is now known as paramite | 15:01 | |
paramite | jistr, "paramite have you had success with it in the end?" >>> not yet, introspection did not work and I'm trying to make rsync to work now | 15:03 |
jistr | paramite: for introspection the thing i pasted yesterday should do the trick | 15:04 |
jistr | spredzy: sorry! :)) i got carried away by another thing and totally forgot to paste it for you | 15:04 |
jistr | spredzy: https://github.com/rdo-management/inlunch/blob/62bd434381392814450e201203d77b4baf802ac1/answers.yml.example#L106-L110 | 15:04 |
spredzy | jistr, I need to run that and rebuild the images correct ? | 15:05 |
paramite | jistr, ack, it fixed the issue thx | 15:05 |
jistr | spredzy: that's not necessary. just re-run the introspection after that | 15:06 |
*** sdake has joined #tripleo | 15:06 | |
egafford | d0ugal: Responded to your -1 on https://review.openstack.org/#/c/221418/. It's possible I'm missing something (and if I am, please let me know,) but I think that the line you're objecting to is in keeping with handling of pwd fields for other services. Thoughts? | 15:08 |
*** mkovacik has quit IRC | 15:09 | |
*** mkovacik has joined #tripleo | 15:09 | |
dprince | slagle: was there a rush to get the tripleo-common spec in? | 15:09 |
d0ugal | egafford: oh, so it is. sorry. They should probably all be removed at some point | 15:09 |
* dprince would rather see the docker specs land soon too | 15:10 | |
egafford | d0ugal: NP; your comment totally makes sense. | 15:10 |
d0ugal | egafford: I'll re-review now. | 15:10 |
*** sdake_ has quit IRC | 15:10 | |
egafford | d0ugal: I fully support a later revision patch to remove the lot of them: they are effectively replacing unset with None. | 15:10 |
d0ugal | +1 | 15:10 |
dprince | slagle: gotta say, with all the restructuring upstream it still feels a bit early to get the API spec right for tripleo-common. Not saying that work on it can't proceed, just that it is likely to change perhaps... | 15:11 |
*** tosky has quit IRC | 15:13 | |
*** Marga_ has joined #tripleo | 15:14 | |
*** social has joined #tripleo | 15:14 | |
*** jerrygb has joined #tripleo | 15:19 | |
*** jerrygb has quit IRC | 15:19 | |
*** jerrygb has joined #tripleo | 15:19 | |
*** shardy_ has joined #tripleo | 15:22 | |
*** shardy has quit IRC | 15:24 | |
*** shardy_ has quit IRC | 15:27 | |
*** shardy has joined #tripleo | 15:28 | |
* mkovacik is away: I'm busy | 15:32 | |
*** tosky has joined #tripleo | 15:35 | |
*** tosky has quit IRC | 15:35 | |
*** tosky has joined #tripleo | 15:35 | |
*** persia has quit IRC | 15:38 | |
*** paramite has quit IRC | 15:44 | |
*** Marga_ has quit IRC | 15:44 | |
*** Marga_ has joined #tripleo | 15:45 | |
*** alop has joined #tripleo | 15:49 | |
*** tzumainn has quit IRC | 15:53 | |
*** lazy_prince has joined #tripleo | 15:53 | |
dprince | shardy: just one additional sentence here and I'm happy I think https://review.openstack.org/#/c/221811/5 | 16:01 |
Slower | dprince: are we back on fedora 22 again? | 16:03 |
dprince | Slower: I'm not, why would we? | 16:03 |
dprince | Slower: did I miss something? | 16:03 |
Slower | wtf | 16:03 |
* dprince answers Slower's question w/ 2 questions | 16:03 | |
Slower | 52f27d73c61d dprince/heat-docker-agents "/usr/bin/os-collect-" 5 minutes ago Up 5 minutes heat-agents | 16:03 |
Slower | [heat-admin@overcloud-novacompute-0 ~]$ sudo docker exec -ti heat-agents /bin/bash | 16:04 |
Slower | [root@overcloud-novacompute-0 /]# cat /etc/redhat-release | 16:04 |
Slower | Fedora release 22 (Twenty Two) | 16:04 |
Slower | maybe I messed something up.. | 16:04 |
gfidente | I think the idea of naming 22 Twenty Two was the brightest naming ever | 16:04 |
gfidente | my 2 cents | 16:04 |
dprince | you should be using dprince/heat-docker-agents-centos | 16:05 |
Slower | dprince: doh! | 16:05 |
Slower | well isn't that grand.. how did that happen? | 16:05 |
Slower | ok | 16:05 |
dprince | Slower: sounds like someone needs to work on rebase skills? | 16:05 |
Slower | heh, could be! | 16:06 |
dprince | Slower: to be fair this patch has evolved a good bit over the past 6 months or so | 16:06 |
Slower | I'm still on devtest too so.. | 16:06 |
Slower | I have some manual changes to make to make it work | 16:06 |
*** jaosorior has quit IRC | 16:07 | |
shardy | Slower: FYI there's a nice new script if you decide to build a new env: | 16:08 |
shardy | https://github.com/openstack/tripleo-common/blob/master/scripts/tripleo.sh | 16:08 |
dprince | Slower: it is time to pay the price man | 16:08 |
Slower | ugh, I hate prices.. :) | 16:08 |
Slower | or paying? | 16:09 |
Slower | something.. :) | 16:09 |
dprince | Slower: devtest is banned in tokyo | 16:09 |
shardy | Slower: tl;dr - follow the tripleo-docs virt instructions until instack-virt-setup, ssh onto the undercloud, then run each step of tripleo.sh (the usage is in order) | 16:09 |
shardy | Slower: personally I skipped introspection but that may work now, not sure | 16:09 |
Slower | shardy: ok cool | 16:09 |
slagle | dprince: i dont think it was a rush to get it in. but it's been around for a month. | 16:10 |
slagle | dprince: that's not necessarily long or short, as far as specs go | 16:10 |
dprince | slagle: fair, I'm just concerned it early for that yet | 16:10 |
*** killer_prince has joined #tripleo | 16:10 | |
shardy | dprince: thanks, I'll update it - it's absolutely my intention that nothing changes except we support some extra branches | 16:12 |
shardy | so "moving" was probably the wrong word to use there | 16:12 |
dprince | shardy: perhaps just me being defensive | 16:12 |
dprince | rhallisey: what did the kernel panic man have to say today | 16:12 |
rhallisey | dprince, I'm going to walk over to his desk today | 16:13 |
dprince | rhallisey: you should take a posse | 16:13 |
slagle | dprince: i can agree that it might be too early to say we know what the implementation will be exactly, i think we got about specific as we felt comfortable with in the spec | 16:13 |
rhallisey | just want to be sure compute node is set | 16:13 |
*** lazy_prince has quit IRC | 16:14 | |
slagle | dprince: but, it's not too early to start the actual work on solving it, imo anyway | 16:14 |
dprince | slagle: agree, part of my wishes we had more than 1 implementation to deal with for tis | 16:14 |
dprince | this | 16:14 |
*** trown is now known as trown|lunch | 16:15 | |
dprince | slagle: that would keep us honest I think | 16:15 |
slagle | dprince: indeed | 16:16 |
slagle | dtantsur: so for the virt power key for ironic, the way it used to work is we had a ssh key on the filesystem. in ironic.conf, there was a configuration value set to the path to that key | 16:20 |
*** jaosorior has joined #tripleo | 16:20 | |
slagle | dtantsur: and that's the key that got used for pxe_ssh nodes. then it changed to be specified on the node itself, and the key saved in the ironic db | 16:20 |
slagle | dtantsur: it just sounded like what you were suggesting is how it used to work, and i assumed was deprecated now | 16:21 |
dtantsur | slagle, no, I'm just pointing that now you can save a path to the key on a node | 16:21 |
dtantsur | instead of saving the key contents | 16:21 |
slagle | ah, i didn't know you could do that | 16:21 |
*** yamahata has joined #tripleo | 16:27 | |
*** jistr has quit IRC | 16:29 | |
*** dtantsur is now known as dtantsur|afk | 16:29 | |
*** lazy_prince has joined #tripleo | 16:31 | |
*** lazy_prince has quit IRC | 16:32 | |
*** killer_prince has quit IRC | 16:33 | |
*** lazy_prince has joined #tripleo | 16:34 | |
*** devvesa has quit IRC | 16:35 | |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Pin delorean to a pre mock version https://review.openstack.org/230011 | 16:37 |
*** zaneb has quit IRC | 16:38 | |
derekh | That may be needed soon ^^ | 16:39 |
*** mgrohar has joined #tripleo | 16:42 | |
*** mcornea has joined #tripleo | 16:43 | |
*** mgrohar has quit IRC | 16:44 | |
*** trown|lunch is now known as trown | 16:56 | |
*** persia has joined #tripleo | 16:57 | |
*** derekh has quit IRC | 17:01 | |
*** sdake_ has joined #tripleo | 17:02 | |
*** morazi has quit IRC | 17:03 | |
*** sdake_ has quit IRC | 17:04 | |
*** jerrygb has quit IRC | 17:04 | |
*** sdake has quit IRC | 17:05 | |
*** Marga_ has quit IRC | 17:07 | |
*** Marga_ has joined #tripleo | 17:08 | |
*** gfidente has quit IRC | 17:11 | |
*** Marga_ has quit IRC | 17:13 | |
openstackgerrit | James Slagle proposed openstack/tripleo-docs: Document tripleo.sh https://review.openstack.org/230027 | 17:17 |
*** shivrao has joined #tripleo | 17:19 | |
*** tosky has quit IRC | 17:21 | |
*** morazi has joined #tripleo | 17:22 | |
*** athomas has quit IRC | 17:32 | |
*** akrivoka has quit IRC | 17:35 | |
*** Nisha_away has joined #tripleo | 17:35 | |
*** Marga_ has joined #tripleo | 17:36 | |
*** jaosorior has quit IRC | 17:38 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: Deprecate the undercloud templates. https://review.openstack.org/229893 | 17:41 |
*** jaosorior has joined #tripleo | 17:42 | |
*** rhallisey has quit IRC | 17:50 | |
*** mcornea has quit IRC | 17:50 | |
*** rhallisey has joined #tripleo | 17:50 | |
*** rhallisey has quit IRC | 17:50 | |
*** rhallisey has joined #tripleo | 17:51 | |
*** pelix has quit IRC | 17:51 | |
*** dprince has quit IRC | 17:54 | |
*** rhallisey has quit IRC | 18:00 | |
*** rhallisey_ has joined #tripleo | 18:01 | |
*** lazy_prince has quit IRC | 18:02 | |
*** CheKoLyN has joined #tripleo | 18:03 | |
*** shardy is now known as shardy_afk | 18:08 | |
*** BharatK has quit IRC | 18:09 | |
*** dprince has joined #tripleo | 18:09 | |
dsneddon | dprince, EC2MetadataIp and ControlPlaneDefaultRoute have no default values, so you can no longer deploy a stack with just network-isolation.yaml and net-single-nic-with-vlans.yaml, you need a network-environment.yaml file now: https://bugzilla.redhat.com/show_bug.cgi?id=1268070 | 18:22 |
openstack | bugzilla.redhat.com bug 1268070 in openstack-tripleo-heat-templates "EC2MetadataIp and ControlPlaneDefaultRoute Have No Default Value" [Unspecified,New] - Assigned to jason.dobies | 18:22 |
*** rhallisey_ has quit IRC | 18:23 | |
dsneddon | dprince, Rather than set default values, what do you think about creating a sample network-environment.yaml inside of environments/, that would have default values and serve as an example? | 18:23 |
dsneddon | dprince, If you like that idea, I'll submit a patch | 18:23 |
dprince | dsneddon: values for those are required though | 18:25 |
dprince | dsneddon: and we don't have a good default | 18:25 |
dsneddon | dprince, 192.0.2.1 is a good default for both, no? | 18:25 |
dprince | dsneddon: I don't think so | 18:25 |
*** penick has joined #tripleo | 18:25 | |
dsneddon | dprince, Hmm, then what values would I use there if my undercloud is 192.0.2.1? | 18:25 |
dprince | dsneddon: I think it is entirely likely that you might cause someone's environment to fail if you set these by default | 18:25 |
dsneddon | dprince, I'm not suggesting we set those as defaults. I'm suggesting that we have an example network-environment.yaml that sets those defaults. | 18:26 |
dprince | dsneddon: I use my actual router | 18:26 |
dprince | dsneddon: so these | 18:26 |
dprince | ControlPlaneDefaultRoute: 172.19.0.1 | 18:26 |
dprince | EC2MetadataIp: 172.19.0.3 | 18:26 |
dprince | DnsServers: ['8.8.8.8'] | 18:26 |
dprince | dsneddon: 172.19.0.1 is my router | 18:26 |
dprince | 172.19.0.3 is my instack VM (seed) | 18:26 |
dsneddon | dprince, Right, but if we don't even have an example of how to set those values, then it's a mystery that is only answered in docs. | 18:27 |
dprince | dsneddon: totally fine with the example, perhaps you could add it to network isolation then? | 18:27 |
*** rhallisey has joined #tripleo | 18:27 | |
dprince | dsneddon: since they are required? | 18:27 |
dprince | dsneddon: FWIW I tried making the EC2MetadataIp parameter go away (with a reasoable default). | 18:28 |
dsneddon | dprince, Yeah, I know. A valiant effort. | 18:28 |
dprince | not everyone was pleased so I nuked it | 18:28 |
jaosorior | exit | 18:29 |
*** jaosorior has quit IRC | 18:29 | |
dprince | I actually think we might be able to "harvest" the Ec2MetadataIp route via os-net-config | 18:29 |
dprince | and thus avoid setting it up in template land | 18:29 |
dprince | dsneddon: ^ | 18:29 |
dsneddon | dprince, Interesting. Where would we get that value, from DHCP? | 18:29 |
dprince | dsneddon: yes, on the first run DHCP has it | 18:30 |
dsneddon | dprince, That's a good idea. | 18:30 |
dprince | dsneddon: again, perhaps a bit of trickery... so I'm not sure everyone would be happy | 18:30 |
dsneddon | dprince, By the way, on another topic, did you see the reviews for Linux bonds and bridges? It looks like we need additional packages on the CI boxes to make that work. Is there a way to include those dependencies in the os-net-config spec? | 18:31 |
*** openstackgerrit has quit IRC | 18:31 | |
dsneddon | dprince, ebtables and bridge-utils is what is missing | 18:31 |
*** openstackgerrit has joined #tripleo | 18:31 | |
dprince | dsneddon: sure, submit them here https://github.com/openstack-packages/os-net-config | 18:31 |
dprince | dsneddon: git review... perhaps mark it WIP until we actually land those patches though | 18:32 |
dsneddon | dprince, Will do, thanks. | 18:32 |
*** Nisha_away has quit IRC | 18:46 | |
*** julim has quit IRC | 18:48 | |
*** rhallisey has quit IRC | 18:55 | |
*** rhallisey has joined #tripleo | 18:56 | |
*** rhallisey has quit IRC | 18:56 | |
dprince | slagle: this patch series looks super useful https://review.openstack.org/#/c/213902/ | 18:57 |
*** hashar has joined #tripleo | 19:03 | |
*** rhallisey has joined #tripleo | 19:14 | |
rhallisey | dprince, I think the kernel panic issue has been solved already | 19:15 |
rhallisey | will 2x check. Going to try in rhel atomic 7.1 | 19:15 |
dprince | rhallisey: got a BZ? are we just waiting on a kernel update then? | 19:15 |
rhallisey | centos has an old kernel in the version we're using | 19:15 |
rhallisey | https://bugzilla.redhat.com/show_bug.cgi?id=1268047 | 19:15 |
openstack | rhallisey: Error: Error getting bugzilla.redhat.com bug #1268047: NotPermitted | 19:15 |
dprince | rhallisey: so it is an issue with the VMs kernel? | 19:15 |
*** sthillma has joined #tripleo | 19:15 | |
rhallisey | dprince, ya | 19:16 |
dprince | rhallisey: aha, lets bump it then | 19:16 |
rhallisey | dprince, supposedly this is fixed in rhel and I'm going to check that | 19:16 |
dprince | rhallisey: cool | 19:16 |
rhallisey | dprince, he immediately knew it was docker | 19:17 |
*** david-lyle has quit IRC | 19:18 | |
rhallisey | now just need to figure out the neutron-agent issue | 19:18 |
*** zaneb has joined #tripleo | 19:18 | |
dprince | rhallisey: docker on the host? | 19:18 |
openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Fix pep8 errors in tripleo-jobs.py https://review.openstack.org/230091 | 19:20 |
*** matbu_ has quit IRC | 19:23 | |
*** matbu has joined #tripleo | 19:25 | |
Slower | rhallisey: dprince: cool! | 19:25 |
*** david-lyle has joined #tripleo | 19:27 | |
*** shardy_afk is now known as shardy | 19:33 | |
*** julim has joined #tripleo | 19:33 | |
*** jprovazn has quit IRC | 19:52 | |
*** ifarkas has quit IRC | 19:55 | |
*** paramite has joined #tripleo | 19:58 | |
*** lblanchard has quit IRC | 19:59 | |
*** jtomasek has joined #tripleo | 20:10 | |
openstackgerrit | Lokesh Jain proposed openstack/tripleo-heat-templates: Changes for configuring Nuage https://review.openstack.org/230116 | 20:11 |
*** tiswanso has quit IRC | 20:12 | |
Slower | dprince: it looks like the i/o error is another kernel bug | 20:15 |
*** paramite is now known as paramite|afk | 20:16 | |
Slower | dprince: http://fpaste.org/273814/73056414/ | 20:16 |
openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: Docker compute role configured via Puppet https://review.openstack.org/209505 | 20:26 |
rhallisey | dprince, let's use f22 atomic | 20:27 |
rhallisey | centos kernel is giving us grief | 20:27 |
Slower | dprince: I think docker is pretty young and the 'new hotness', there's been a lot of bug fixes going on to support it | 20:31 |
Slower | dprince: I think it may be more prudent to use the latest and greatest in this situation | 20:31 |
*** dsneddon is now known as dsneddon_biab | 20:34 | |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Destroy previous test env VM's earlier https://review.openstack.org/230129 | 20:37 |
*** derekh has joined #tripleo | 20:46 | |
*** hashar has quit IRC | 20:47 | |
openstackgerrit | imain proposed openstack/tripleo-heat-templates: Docker compute role configured via Puppet https://review.openstack.org/209505 | 20:48 |
derekh | A commit to switch delorean from using docker to mock could merge at any stage soon, our CI isn't yet ready to use mock, | 20:48 |
derekh | I'll work on it from tomorrow but in the meantime we may want to pin delorean | 20:49 |
derekh | https://review.openstack.org/#/c/230011/1 | 20:49 |
derekh | dprince: shardy slagle ^ | 20:49 |
slagle | derekh: thanks, sounds good to me | 20:49 |
slagle | i just +2'd | 20:49 |
derekh | slagle: thanks | 20:49 |
slagle | i had that open earlier in a tab, but forgot to review | 20:49 |
derekh | slagle: cool, I had a -1 on it until just now anyways | 20:51 |
slagle | oh, maybe that was why :) i don't remember 4 hours ago honestly | 20:52 |
derekh | slagle: 4 hours is a long time in tripleo | 20:53 |
shardy | derekh: done | 20:54 |
openstackgerrit | Merged openstack-infra/tripleo-ci: Pin delorean to a pre mock version https://review.openstack.org/230011 | 20:54 |
derekh | shardy: btw, I finally got CI working today generating packages with tripleo.sh https://review.openstack.org/#/c/229425/ | 20:55 |
*** rpothier has quit IRC | 20:55 | |
shardy | derekh: nice! I'll take a look in the morning :) | 20:55 |
derekh | shardy: cool, I'm trying to decide which is better put the same delorean pin in tripleo.sh now before we merge it | 20:56 |
derekh | shardy: or hold off and use the mock version instead | 20:56 |
derekh | shardy: sure take a look in the morning and we can figure out whats best | 20:56 |
openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: loadbalancer: fix MySQL timeout HAproxy config https://review.openstack.org/223223 | 20:57 |
jpeeler | rhallisey: i just got the kernel panic again, guess i'll switch too | 20:58 |
rhallisey | jpeeler, ya there's a double kernel panic here | 20:58 |
rhallisey | centos atomic has has to old of a kernel | 20:58 |
rhallisey | docker blows it up | 20:58 |
jpeeler | ah, well i'm seeing it on centos (not atomic) | 20:59 |
jpeeler | probably the same is true there though | 20:59 |
rhallisey | jpeeler, I just updated the compute patch. We will use fedora atomic | 20:59 |
shardy | jpeeler: I had my centos7 undercloud kernel panic earlier | 20:59 |
rhallisey | shardy, with containers right? | 20:59 |
*** untriaged-bot has joined #tripleo | 21:00 | |
untriaged-bot | Untriaged bugs so far: | 21:00 |
untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1499298 | 21:00 |
openstack | Launchpad bug 1499298 in tripleo "Installers should use pymysql db driver as of Liberty" [Undecided,New] | 21:00 |
untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1501378 | 21:00 |
shardy | rhallisey: I may have been deploying the docker overcloud, but I wasn't running docker on the undercloud | 21:00 |
openstack | Launchpad bug 1501378 in tripleo "pacemaker neutron resource constraints and cleanup" [Undecided,Triaged] | 21:00 |
untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1501296 | 21:00 |
openstack | Launchpad bug 1501296 in tripleo "Stack updates fail due to a new resource type" [Medium,New] | 21:00 |
*** untriaged-bot has quit IRC | 21:00 | |
Slower | we have 2 different kernel issues | 21:00 |
Slower | both with centos | 21:00 |
jpeeler | is there somewhere we should be sending this reports? | 21:00 |
*** derekh has quit IRC | 21:00 | |
Slower | instack seed VM panics and crashes | 21:00 |
shardy | totally wedged the box tho, I had to reset it | 21:00 |
Slower | compute node has XFS i/o errors causing the container to fail | 21:00 |
rhallisey | jpeeler, I'm on it. They are aware and have supposedly fixed this in a later kernel | 21:00 |
Slower | shardy: you had to reset the HW box? | 21:01 |
shardy | Slower: no, it was a VM | 21:01 |
jpeeler | ok, guess i'll just reset and go. this makes working pretty difficult - all i did was one redeploy | 21:01 |
Slower | shardy: right, I thought it was ironic causing it? | 21:01 |
rhallisey | the issue is propagating down | 21:02 |
Slower | which is also wierd :) | 21:02 |
Slower | it shouldn't be able to propogate through a vm? | 21:02 |
shardy | I don't see how it can be container related, I didn't have containers running on the host or on the undercloud, only in a VM on the host | 21:02 |
Slower | shardy: agreed | 21:03 |
shardy | I didn't investigate too much tho tbh | 21:03 |
dprince | Slower: what if we make it so that when os-net-config ifdown's eth0 to reconfigure it (the first time) it uses nmcli if networkmanager is enabled? | 21:04 |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: neutron: enable nova-event-callback by default https://review.openstack.org/225962 | 21:04 |
rhallisey | kernel panic logs -> http://fpaste.org/272780/44354982/ | 21:04 |
rhallisey | line 62 | 21:04 |
rhallisey | when I asked the kernel guy | 21:04 |
rhallisey | he said that is from docker | 21:04 |
rhallisey | the vm failure propagates down through instack and kills it too | 21:05 |
rhallisey | even though docker is only in a vm | 21:05 |
Slower | does ironic use device mapper? | 21:05 |
Slower | dprince: that could work.. | 21:06 |
shardy | rhallisey: That is pretty scary | 21:06 |
Slower | dprince: did you see the new centos issue tho? | 21:06 |
rhallisey | shardy, it is supposed to be fixed though | 21:06 |
dprince | Slower: no, bug/link? | 21:06 |
Slower | dprince: http://fpaste.org/273814/73056414/ | 21:07 |
Slower | dprince: that's the source of our i/o error we're seeing on the neutron container | 21:07 |
Slower | dprince: I'm getting the sense that docker has moved pretty fast and exposed a lot of bugs.. | 21:07 |
rhallisey | https://bugzilla.redhat.com/show_bug.cgi?id=1268047 | 21:07 |
openstack | rhallisey: Error: Error getting bugzilla.redhat.com bug #1268047: NotPermitted | 21:07 |
dprince | Slower: can we pin to a previous version? | 21:07 |
rhallisey | ^ kernel bz | 21:08 |
Slower | dprince: and I wonder if we wouldn't be better off using newer upstream | 21:08 |
*** jayg is now known as jayg|g0n3 | 21:08 | |
Slower | dprince: of docker? We actually need the latest.. | 21:08 |
dprince | Slower: or upstream is fine too | 21:08 |
Slower | dprince: by that I mean fedora atomic instead of centos (which has older kernel) | 21:08 |
dprince | Slower: we can move back to Fedora atomic for now. Fine w/ me | 21:09 |
dprince | Slower: just update the README is all that is required there | 21:09 |
Slower | dprince: ok good.. cause we already pushed that change ;-) | 21:09 |
dprince | Slower: great, press ahead man | 21:09 |
dprince | Slower: do you want me to look into the potential around using nmcli in os-net-config? | 21:09 |
Slower | honestly if we are on fedora atomic I think we can just disable NM and it'll work.. | 21:10 |
dprince | Slower: would that help resolve the DHCP lease issue w/ NetworkManager? | 21:10 |
Slower | but we may need that for rhel, I'm not sure | 21:10 |
dprince | Slower: okay, we may need to revisit this. | 21:10 |
Slower | dprince: I'll try it out soon (once I get instack going) and let you know | 21:10 |
dprince | Slower: might be good to document this "idea", cause I will surely forget it | 21:11 |
Slower | dprince: you just want a comment someplace, or? | 21:11 |
Slower | maybe in the etherpad | 21:12 |
*** dsneddon_biab is now known as dsneddon | 21:12 | |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: puppet: run keystone in wsgi https://review.openstack.org/213175 | 21:12 |
dprince | Slower: sure, I'll leave a note in the etherpad | 21:12 |
*** jpeeler has quit IRC | 21:12 | |
dprince | Slower: even though we are hitting new errors I'm not sure we should block this heat template patch for those | 21:13 |
Slower | dprince: I totally agree | 21:14 |
dprince | Slower: getting that landed would probably help more people get involved, so perhaps just organize all the issues as best you can and maybe we can convince shardy it is okay even if it doesn't fully work today | 21:14 |
Slower | it's actually a bit confusing as it is, we're all making changes to a patch with no diff tracking.. | 21:14 |
Slower | it's gonna work by tomorrow darnit! | 21:15 |
Slower | :) | 21:15 |
rhallisey | It should work now | 21:15 |
Slower | never closer! | 21:15 |
rhallisey | was working for awhile now, but these kernel bugs came outa no where | 21:15 |
rhallisey | ahhh | 21:15 |
*** paramite|afk has quit IRC | 21:16 | |
Slower | no they came from centos :) | 21:18 |
*** Goneri has quit IRC | 21:19 | |
*** rhallisey has quit IRC | 21:23 | |
*** shardy has quit IRC | 21:26 | |
*** Marga_ has quit IRC | 21:28 | |
*** trown is now known as trown|outttypeww | 21:30 | |
*** Marga_ has joined #tripleo | 21:30 | |
*** david-lyle has quit IRC | 21:43 | |
*** rlandy has quit IRC | 21:43 | |
*** weshay has quit IRC | 21:44 | |
*** Marga_ has quit IRC | 21:46 | |
*** Marga_ has joined #tripleo | 21:47 | |
*** spzala has quit IRC | 21:54 | |
*** spzala has joined #tripleo | 21:55 | |
*** david-lyle has joined #tripleo | 21:56 | |
*** Marga_ has quit IRC | 21:59 | |
*** Marga_ has joined #tripleo | 21:59 | |
*** CheKoLyN has quit IRC | 21:59 | |
*** Marga_ has quit IRC | 22:00 | |
*** Marga_ has joined #tripleo | 22:00 | |
*** nijaba has quit IRC | 22:16 | |
*** Goneri has joined #tripleo | 22:20 | |
*** egafford has left #tripleo | 22:20 | |
*** nijaba has joined #tripleo | 22:26 | |
*** nijaba has quit IRC | 22:26 | |
*** nijaba has joined #tripleo | 22:26 | |
*** nijaba has quit IRC | 22:33 | |
*** nijaba has joined #tripleo | 22:36 | |
*** nijaba has quit IRC | 22:36 | |
*** nijaba has joined #tripleo | 22:36 | |
*** sthillma_ has joined #tripleo | 22:40 | |
*** sthillma has quit IRC | 22:43 | |
*** sthillma_ is now known as sthillma | 22:43 | |
*** dprince has quit IRC | 22:46 | |
*** Goneri has quit IRC | 22:47 | |
*** hashar has joined #tripleo | 22:50 | |
*** pradk has quit IRC | 22:59 | |
*** mudassirlatif has joined #tripleo | 23:01 | |
mudassirlatif | Hi, newbie question: How do I find out if tripleo release has been cut? Is it too late for change anything for Liberty? If not, what are the deadlines? | 23:03 |
*** chlong has quit IRC | 23:07 | |
*** hashar has quit IRC | 23:08 | |
*** sthillma_ has joined #tripleo | 23:16 | |
*** sthillma has quit IRC | 23:19 | |
*** sthillma_ is now known as sthillma | 23:19 | |
*** alop has quit IRC | 23:25 | |
*** rasca_ has joined #tripleo | 23:26 | |
*** rasca has quit IRC | 23:29 | |
*** rhallisey has joined #tripleo | 23:29 | |
openstackgerrit | Derek Higgins proposed openstack/diskimage-builder: Ignore ./proc when deleting python object files https://review.openstack.org/230202 | 23:31 |
*** Goneri has joined #tripleo | 23:56 | |
*** bnemec has joined #tripleo | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!