*** yuanying has quit IRC | 00:09 | |
*** yuanying has joined #tripleo | 00:16 | |
*** yuanying has quit IRC | 00:19 | |
*** yuanying has joined #tripleo | 00:22 | |
*** chlong has joined #tripleo | 00:40 | |
*** eil397 has quit IRC | 01:04 | |
*** shivrao has quit IRC | 01:06 | |
*** thrash|f00dz is now known as thrash|g0ne | 01:09 | |
*** yuanying has quit IRC | 01:11 | |
*** yuanying has joined #tripleo | 01:12 | |
*** stendulker has joined #tripleo | 01:13 | |
*** rwsu has joined #tripleo | 01:14 | |
*** dsneddon_ has quit IRC | 01:19 | |
*** dsneddon_ has joined #tripleo | 01:19 | |
*** yuanying has quit IRC | 01:21 | |
*** yuanying has joined #tripleo | 01:23 | |
*** crinkle_ has joined #tripleo | 01:30 | |
*** crinkle has quit IRC | 01:31 | |
*** Hunner has quit IRC | 01:31 | |
*** crinkle_ is now known as crinkle | 01:31 | |
*** penick has quit IRC | 01:31 | |
*** Hunner has joined #tripleo | 01:36 | |
*** Hunner has joined #tripleo | 01:36 | |
*** tiswanso has joined #tripleo | 01:36 | |
*** alop has quit IRC | 01:37 | |
*** weshay_xchat has joined #tripleo | 01:38 | |
*** tiswanso has quit IRC | 01:43 | |
*** tiswanso has joined #tripleo | 01:43 | |
*** weshay_xchat is now known as weshay | 01:48 | |
*** tzumainn has quit IRC | 02:01 | |
*** stendulker has quit IRC | 02:25 | |
*** yamahata has quit IRC | 02:39 | |
*** shivrao has joined #tripleo | 02:46 | |
*** shivrao has quit IRC | 02:59 | |
*** weshay has quit IRC | 03:08 | |
*** yuanying has quit IRC | 03:15 | |
*** weshay_xchat has joined #tripleo | 03:18 | |
*** yuanying has joined #tripleo | 03:19 | |
*** yuanying has quit IRC | 03:24 | |
*** yuanying has joined #tripleo | 03:35 | |
*** stendulker has joined #tripleo | 03:38 | |
*** rbrady has quit IRC | 03:40 | |
*** weshay_xchat has quit IRC | 03:41 | |
*** yuanying_ has joined #tripleo | 03:41 | |
*** yuanying has quit IRC | 03:44 | |
*** links has joined #tripleo | 03:49 | |
*** yuanying_ has quit IRC | 04:06 | |
*** yuanying has joined #tripleo | 04:07 | |
*** yuanying has quit IRC | 04:08 | |
*** yuanying has joined #tripleo | 04:08 | |
*** rbrady has joined #tripleo | 04:08 | |
*** ryansb has quit IRC | 04:11 | |
*** ryansb has joined #tripleo | 04:11 | |
*** ryansb has joined #tripleo | 04:11 | |
*** Ng has quit IRC | 04:11 | |
*** SlickNik has quit IRC | 04:11 | |
*** SlickNik has joined #tripleo | 04:12 | |
*** trown|outttypeww has quit IRC | 04:12 | |
*** Nakato has quit IRC | 04:12 | |
*** trown has joined #tripleo | 04:12 | |
*** jroll has quit IRC | 04:13 | |
*** gus has quit IRC | 04:13 | |
*** davidlenwell_ has joined #tripleo | 04:13 | |
*** davidlenwell has quit IRC | 04:13 | |
*** chlong has quit IRC | 04:13 | |
*** Nakato has joined #tripleo | 04:13 | |
*** jpeeler has quit IRC | 04:13 | |
*** bandini has quit IRC | 04:13 | |
*** bandini has joined #tripleo | 04:14 | |
*** gus has joined #tripleo | 04:14 | |
*** jroll has joined #tripleo | 04:14 | |
*** pblaho has quit IRC | 04:19 | |
*** pblaho has joined #tripleo | 04:19 | |
*** jrist has quit IRC | 04:19 | |
*** cmyster has quit IRC | 04:19 | |
*** Ng has joined #tripleo | 04:20 | |
*** ChanServ sets mode: +v Ng | 04:20 | |
*** jrist has joined #tripleo | 04:20 | |
*** jrist has joined #tripleo | 04:20 | |
*** cmyster has joined #tripleo | 04:20 | |
*** xinwu has quit IRC | 04:20 | |
*** jpeeler has joined #tripleo | 04:21 | |
*** xinwu has joined #tripleo | 04:21 | |
*** xinwu has quit IRC | 04:21 | |
*** shivrao has joined #tripleo | 04:21 | |
*** xinwu has joined #tripleo | 04:23 | |
*** xinwu has quit IRC | 04:26 | |
*** chlong has joined #tripleo | 04:28 | |
openstackgerrit | Merged openstack/diskimage-builder: Print unparsable file in pkg-map https://review.openstack.org/267305 | 04:47 |
---|---|---|
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Fix tar listing in functional tests https://review.openstack.org/276037 | 04:51 |
*** jtomasek_ has joined #tripleo | 05:02 | |
*** davidlenwell_ has quit IRC | 05:02 | |
*** davidlenwell has joined #tripleo | 05:05 | |
*** ChanServ sets mode: +v davidlenwell | 05:05 | |
*** akuznetsov has joined #tripleo | 05:12 | |
*** xinwu has joined #tripleo | 05:20 | |
*** anande has joined #tripleo | 05:20 | |
*** anande has quit IRC | 05:27 | |
*** akuznetsov has quit IRC | 05:42 | |
*** oshvartz has quit IRC | 05:50 | |
*** masco has joined #tripleo | 06:05 | |
*** akuznetsov has joined #tripleo | 06:05 | |
*** penick has joined #tripleo | 06:13 | |
*** tiswanso has quit IRC | 06:16 | |
*** penick_ has joined #tripleo | 06:16 | |
*** penick has quit IRC | 06:18 | |
*** penick_ is now known as penick | 06:18 | |
*** akuznetsov has quit IRC | 06:18 | |
*** akuznetsov has joined #tripleo | 06:19 | |
*** akuznetsov has quit IRC | 06:29 | |
*** jtomasek_ has quit IRC | 06:40 | |
*** jprovazn has joined #tripleo | 06:55 | |
*** Marga_ has joined #tripleo | 06:56 | |
*** tzumainn has joined #tripleo | 07:01 | |
*** oshvartz has joined #tripleo | 07:01 | |
*** penick has quit IRC | 07:23 | |
*** dshulyak has joined #tripleo | 07:25 | |
*** akuznetsov has joined #tripleo | 07:32 | |
*** tzumainn has quit IRC | 07:38 | |
*** ukalifon has joined #tripleo | 07:40 | |
*** dsneddon_ has quit IRC | 07:42 | |
*** hjensas has quit IRC | 07:50 | |
*** akuznetsov has quit IRC | 07:55 | |
*** jistr|mobi has joined #tripleo | 07:59 | |
openstackgerrit | Merged openstack/diskimage-builder: Add support for gentoo to simple-init https://review.openstack.org/273790 | 08:00 |
*** jdob has joined #tripleo | 08:01 | |
*** bnemec has joined #tripleo | 08:02 | |
*** ukalifon has quit IRC | 08:04 | |
*** fgimenez has joined #tripleo | 08:06 | |
*** shivrao has quit IRC | 08:06 | |
*** Marga_ has quit IRC | 08:10 | |
*** Marga_ has joined #tripleo | 08:10 | |
*** xinwu has quit IRC | 08:13 | |
*** tosky has joined #tripleo | 08:14 | |
*** akuznetsov has joined #tripleo | 08:15 | |
*** hjensas has joined #tripleo | 08:20 | |
*** hjensas has quit IRC | 08:20 | |
*** hjensas has joined #tripleo | 08:20 | |
*** aufi has joined #tripleo | 08:21 | |
*** dsneddon_ has joined #tripleo | 08:24 | |
*** gfidente has joined #tripleo | 08:24 | |
*** jistr has joined #tripleo | 08:29 | |
*** ifarkas has joined #tripleo | 08:30 | |
*** jcoufal has joined #tripleo | 08:30 | |
*** jaosorior has joined #tripleo | 08:31 | |
*** mkovacik has joined #tripleo | 08:31 | |
*** akuznetsov has quit IRC | 08:31 | |
*** olap has joined #tripleo | 08:32 | |
*** akuznetsov has joined #tripleo | 08:41 | |
*** jtomasek__ has joined #tripleo | 08:43 | |
*** akuznetsov has quit IRC | 08:53 | |
*** liverpooler has joined #tripleo | 08:53 | |
*** jcoufal has quit IRC | 08:54 | |
*** bvandenh has joined #tripleo | 08:56 | |
*** rhallisey has joined #tripleo | 09:04 | |
*** mkovacik has quit IRC | 09:06 | |
*** paramite has joined #tripleo | 09:10 | |
*** paramite is now known as paramite|afk | 09:10 | |
*** jistr has quit IRC | 09:16 | |
*** openstackgerrit has quit IRC | 09:17 | |
*** openstackgerrit has joined #tripleo | 09:17 | |
*** paramite|afk is now known as paramite | 09:21 | |
*** dsneddon_ has quit IRC | 09:22 | |
*** mbound has joined #tripleo | 09:24 | |
*** devvesa has joined #tripleo | 09:24 | |
devvesa | hi lucasagomes: you are the ironic master, right? | 09:27 |
*** nico_auv has joined #tripleo | 09:27 | |
devvesa | I have a question for you | 09:29 |
*** dsneddon_ has joined #tripleo | 09:29 | |
*** ukalifon1 has joined #tripleo | 09:30 | |
*** mcornea has joined #tripleo | 09:32 | |
devvesa | or maybe trown... | 09:32 |
lucasagomes | devvesa, I wish, what's up? | 09:33 |
devvesa | I have a baremetal environment right now | 09:35 |
devvesa | And I can see all the registered nodes as 'manageable' | 09:35 |
devvesa | But when I run the './tripleo.sh --introspect-nodes' script, it gets stuck | 09:36 |
devvesa | It is safe to run the overcloud? (my first time in a baremetal env) | 09:36 |
devvesa | I ignore what the '--introspect-nodes' does | 09:37 |
lucasagomes | devvesa, right, it get stuck where? Booting the nodes ? | 09:38 |
*** morazi has joined #tripleo | 09:38 | |
devvesa | I really don't know. The nodes were already on | 09:39 |
devvesa | I see the following | 09:39 |
devvesa | http://fpaste.org/318087/14545217/ | 09:39 |
devvesa | And keeps waiting... | 09:39 |
lucasagomes | devvesa, right, do you have access to the console? | 09:40 |
lucasagomes | see if the nodes booted | 09:41 |
*** electrofelix has joined #tripleo | 09:41 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: neutron: delete by default router/dhcp namespaces https://review.openstack.org/271569 | 09:42 |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: neutron: delete by default router/dhcp namespaces https://review.openstack.org/276115 | 09:43 |
*** mgould has joined #tripleo | 09:43 | |
devvesa | lucasagomes: the nodes are up: http://fpaste.org/318317/45790701/ | 09:44 |
lucasagomes | devvesa, that's fine yes... but it may not booted the ramdisk | 09:47 |
lucasagomes | there are a couple of things that can go wrong at that stage: DHCP problems, PXE/iPXE problems, kernel problems | 09:48 |
*** trown has quit IRC | 09:49 | |
devvesa | lucasagomes: I had problems in the Virtual Environment with ramdisk as well... Had to upload the old ramdisk | 09:50 |
devvesa | Maybe you remember | 09:50 |
devvesa | Is there any log that I can look at? | 09:51 |
*** ukalifon1 has quit IRC | 09:53 | |
lucasagomes | devvesa, logs depend of the type of machine you are booting... I mean for virtual you could redirect the console to a file (we do that in gate) | 09:56 |
lucasagomes | for baremetal you have to access the remote controller to see the logs | 09:56 |
lucasagomes | if it fails to boot, the system will eventually time out and error | 09:56 |
lucasagomes | remote console to see the logs* | 09:57 |
devvesa | Ok. I'm there | 09:57 |
*** apetrich has quit IRC | 09:58 | |
openstackgerrit | Merged openstack/instack-undercloud: puppet: fix uchiwa password parameter https://review.openstack.org/274202 | 09:59 |
*** dsneddon_ has quit IRC | 10:01 | |
*** rlandy has quit IRC | 10:04 | |
openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: Deploy Monitoring on the undercloud with Puppet https://review.openstack.org/276127 | 10:05 |
*** rlandy has joined #tripleo | 10:08 | |
*** gfidente has quit IRC | 10:10 | |
*** apetrich has joined #tripleo | 10:12 | |
*** gfidente has joined #tripleo | 10:15 | |
*** gfidente has quit IRC | 10:15 | |
*** gfidente has joined #tripleo | 10:15 | |
*** apetrich has quit IRC | 10:17 | |
*** NobodyCa1 has joined #tripleo | 10:18 | |
*** dsneddon_ has joined #tripleo | 10:19 | |
*** NobodyCa1 has quit IRC | 10:26 | |
*** apetrich has joined #tripleo | 10:32 | |
*** dshulyak_ has joined #tripleo | 10:39 | |
*** dshulyak has quit IRC | 10:39 | |
*** dshulyak_ is now known as dshulyak | 10:39 | |
*** bnemec has quit IRC | 10:40 | |
*** fgimenez has quit IRC | 10:41 | |
*** fgimenez has joined #tripleo | 10:42 | |
*** bnemec has joined #tripleo | 10:42 | |
*** paramite is now known as paramite|afk | 10:45 | |
*** stendulker has quit IRC | 10:55 | |
*** dsneddon_ has quit IRC | 10:56 | |
*** bnemec has quit IRC | 11:01 | |
*** jaosorior has quit IRC | 11:01 | |
*** mgould has quit IRC | 11:04 | |
*** yamahata has joined #tripleo | 11:10 | |
*** paramite|afk is now known as paramite | 11:16 | |
*** mgould has joined #tripleo | 11:18 | |
*** mgould has quit IRC | 11:28 | |
*** mgould has joined #tripleo | 11:32 | |
*** apetrich has quit IRC | 11:34 | |
*** bnemec has joined #tripleo | 11:36 | |
*** panda_ has quit IRC | 11:37 | |
*** panda_ has joined #tripleo | 11:37 | |
*** dsneddon_ has joined #tripleo | 11:37 | |
devvesa | lucasagomes: Downloading the ironic agent from TripleO CI works and I can run introspect-nodes now | 11:44 |
lucasagomes | devvesa, alright. So the previous ramdisk wasn't booting? Did you see anything strange in the console? | 11:44 |
devvesa | the console entered in some kind of recovery mode and keep stuck there | 11:46 |
*** mgould has quit IRC | 11:46 | |
devvesa | Now I have another problem when I deploy the overcloud, though. The nodes stay in 'wait call-back' mode | 11:47 |
*** dsneddon_ has quit IRC | 11:48 | |
devvesa | However, I see the login in the console and I can ping them from the undercloud host | 11:48 |
*** mgould has joined #tripleo | 11:48 | |
devvesa | What is actually ironic waiting? | 11:49 |
*** fgimenez has quit IRC | 11:51 | |
*** bvandenh has quit IRC | 11:51 | |
*** fgimenez has joined #tripleo | 11:53 | |
*** fgimenez has joined #tripleo | 11:53 | |
*** jaosorior has joined #tripleo | 11:55 | |
*** dsneddon_ has joined #tripleo | 11:56 | |
*** jistr has joined #tripleo | 11:56 | |
jaosorior | any +A for this? https://review.openstack.org/#/c/275436/ | 11:57 |
*** apetrich has joined #tripleo | 12:02 | |
*** paramite is now known as paramite|afk | 12:03 | |
*** athomas has joined #tripleo | 12:07 | |
*** apetrich has quit IRC | 12:07 | |
devvesa | @lucasagomes: ^ | 12:11 |
lucasagomes | devvesa, "wait call-back" means that Ironic is waiting for the ramdisk to ping the Ironic API back saying it's ready for deployment | 12:12 |
lucasagomes | devvesa, is it stuck in the login forever? (Cause soon the ironic-python-agent should start and start talking to the ironic api) | 12:12 |
devvesa | lucasagomes: yes, is stuck in the login | 12:13 |
devvesa | I understand that it has already passed the 'ramdisk' phase | 12:13 |
lucasagomes | so "ramdisk" is just the type of image you are booting | 12:14 |
lucasagomes | we run a service called ironic-python-agent when that image boots up, that service will talk to the ironic api | 12:15 |
devvesa | http://i.imgur.com/03hMH1k.png | 12:15 |
lucasagomes | and ironic will then start instructing ipa how to deploy the node (submitting commands to it) | 12:15 |
devvesa | lucasagomes: For what I see, the node has already the CentOS ready to be configured ... | 12:16 |
openstackgerrit | James Slagle proposed openstack/python-tripleoclient: Simplify checking for stack complete https://review.openstack.org/275149 | 12:16 |
lucasagomes | devvesa, oh wait... You sure this booted the ramdisk? It seems that the boot failed and it falled back to the hard disk then | 12:16 |
lucasagomes | try to restart that node please, check the console, see if it's really booting from the network and not from the local disk | 12:16 |
lucasagomes | it may happen that the network boot failed for whatever reason (e.g can't get DHCP) and it fallback'd to the next boot device (which can be the hard disk) | 12:17 |
devvesa | no. it is the node. i've tried a login and now I see the following http://i.imgur.com/ak2Y6aK.png | 12:19 |
devvesa | The IP address of the hostname is the one assigned by neutron-undercloud | 12:19 |
lucasagomes | devvesa, right, so it was deployed?! | 12:20 |
devvesa | yes | 12:20 |
lucasagomes | but ironic is in "wait call-back" ? | 12:21 |
devvesa | yes | 12:21 |
*** paramite|afk is now known as paramite | 12:21 | |
lucasagomes | I don't think it's possible off the top of my head | 12:21 |
lucasagomes | the harddisk of this node was clean before you started the deployment? Or it had an image written on it already ? (from previous deployments) | 12:21 |
devvesa | it had a RedHat | 12:22 |
devvesa | well, I've tried the deployment several times.. so just before it had a CentOS | 12:22 |
devvesa | too | 12:22 |
*** masco has quit IRC | 12:22 | |
lucasagomes | right... yeah cause Ironic will always move the node from "wait call-back" to "deploying" prior to writing the image to the disk | 12:24 |
lucasagomes | devvesa, you can look at the ironic-conductor logs, see if it wrote the image in the disk | 12:24 |
lucasagomes | but I don't think so... I think it couldn't boot via the network and then booted from the disk | 12:24 |
lucasagomes | devvesa, you can restart that node and see if it's failing to boot from the network | 12:25 |
devvesa | ok | 12:25 |
devvesa | lucasagomes: http://i.imgur.com/L06rgbP.png | 12:29 |
devvesa | It doesn't | 12:29 |
lucasagomes | devvesa, alright let it boot then | 12:29 |
devvesa | lucasagomes: same as before. Login screen. 'wait call-back' state | 12:30 |
lucasagomes | devvesa, centos screen? | 12:31 |
devvesa | yes | 12:31 |
lucasagomes | ok... so I may assume that the ironic-python-agent service failed to start on that ramdisk | 12:31 |
lucasagomes | and it's not talking to the ironic API back | 12:31 |
devvesa | And... I can not see the log anywhere, right? | 12:32 |
lucasagomes | devvesa, you can log in that node, if the image was built with the dynamic login element (https://github.com/openstack/diskimage-builder/tree/master/elements/dynamic-login) | 12:33 |
lucasagomes | you can change the httpboot/pxelinux.conf/<mac> and add the parameters there | 12:33 |
lucasagomes | you can also redirect the journald logs to the console so you can see what's going on without logging in | 12:33 |
lucasagomes | just add "systemd.journald.forward_to_console=yes" to the kernel cmdline | 12:34 |
lucasagomes | you can modify the kernel cmdline via ironic's configuration file as well (see pxe_append_parameters config option at ironic.conf) but you will need to restart the ironic-conductor service and deployment to that to get effect | 12:35 |
devvesa | that sounds great | 12:35 |
lucasagomes | devvesa, or you can even try another ramdisk, the upstream one is used in our gate | 12:35 |
lucasagomes | so it's likely that works | 12:35 |
devvesa | by upstream you mean 'master'? | 12:36 |
lucasagomes | devvesa, yup | 12:36 |
lucasagomes | http://tarballs.openstack.org/ironic-python-agent/coreos/files/ | 12:36 |
devvesa | because I took the current one from the CI | 12:36 |
devvesa | in the stable release | 12:36 |
devvesa | (BTW, i'm testing stable/liberty) | 12:36 |
lucasagomes | right, yeah it's backward compatible | 12:37 |
lucasagomes | try the other steps before, if you can login in the ramdisk check for the "ironic-python-agent" service, see if it's started and running | 12:37 |
devvesa | coreos? | 12:38 |
lucasagomes | devvesa, upstream we build it on top of coreos | 12:38 |
*** yamahata has quit IRC | 12:38 | |
openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Add support for --overcloud-update and use it for HA job https://review.openstack.org/260466 | 12:39 |
*** pino|work has quit IRC | 12:42 | |
*** thrash|g0ne is now known as thrash | 12:43 | |
*** weshay_xchat has joined #tripleo | 12:44 | |
*** weshay_xchat is now known as weshay | 12:44 | |
lucasagomes | devvesa, I will grab some food, pelase try to find out whether the service in the ramdisk has started or not and we can continue from there | 12:45 |
devvesa | Thank you very much lucasagomes | 12:45 |
lucasagomes | no problem | 12:45 |
*** lucasagomes is now known as lucas-hungry | 12:46 | |
*** tzumainn has joined #tripleo | 12:49 | |
*** thrash has quit IRC | 12:51 | |
*** aufi_ has joined #tripleo | 12:52 | |
*** aufi has quit IRC | 12:53 | |
*** cmyster has quit IRC | 12:53 | |
*** cmyster has joined #tripleo | 12:54 | |
openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: iptables: add missing rule for ceilometer/ssl https://review.openstack.org/276202 | 12:56 |
*** thrash has joined #tripleo | 12:58 | |
*** thrash has joined #tripleo | 12:58 | |
*** dtantsur|afk is now known as dtantsur | 12:58 | |
openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: iptables: add missing rule for ceilometer/ssl https://review.openstack.org/276202 | 13:02 |
*** aufi_ is now known as aufi | 13:10 | |
*** yamahata has joined #tripleo | 13:12 | |
*** apetrich has joined #tripleo | 13:16 | |
*** jayg|g0n3 is now known as jayg | 13:19 | |
*** apetrich has quit IRC | 13:21 | |
*** yamahata has quit IRC | 13:24 | |
*** bvandenh has joined #tripleo | 13:26 | |
*** trown has joined #tripleo | 13:26 | |
*** yamahata has joined #tripleo | 13:29 | |
*** paramite is now known as paramite|afk | 13:30 | |
*** ukalifon1 has joined #tripleo | 13:32 | |
*** dsneddon_ has quit IRC | 13:34 | |
*** akuznetsov has joined #tripleo | 13:34 | |
*** paramite|afk is now known as paramite | 13:34 | |
*** yamahata has quit IRC | 13:36 | |
*** david-lyle has quit IRC | 13:38 | |
*** gfidente has quit IRC | 13:39 | |
*** ukalifon1 has quit IRC | 13:39 | |
*** apetrich has joined #tripleo | 13:40 | |
*** Goneri has quit IRC | 13:42 | |
*** pradk has joined #tripleo | 13:42 | |
*** trozet has quit IRC | 13:46 | |
*** gfidente has joined #tripleo | 13:47 | |
*** gfidente has quit IRC | 13:47 | |
*** gfidente has joined #tripleo | 13:47 | |
*** dsneddon_ has joined #tripleo | 13:47 | |
*** lucas-hungry is now known as lucasagomes | 13:48 | |
*** paramite is now known as paramite|afk | 13:52 | |
devvesa | lucasagomes: here again? | 13:52 |
lucasagomes | devvesa, yup | 13:52 |
devvesa | I see the following in the console: http://i.imgur.com/p7wYOP7.png | 13:52 |
devvesa | And this in the ironic-api logs: http://fpaste.org/318410/14545938/ | 13:52 |
*** paramite|afk is now known as paramite | 13:53 | |
devvesa | Are both related? | 13:53 |
*** jayg is now known as jayg|g0n3 | 13:53 | |
*** jayg|g0n3 is now known as jayg | 13:53 | |
*** links has quit IRC | 13:56 | |
EmilienM | slagle: https://forge.puppetlabs.com/camptocamp/kmod | 13:56 |
lucasagomes | devvesa, yes!!! lookup() is the first method the ramdisk service will call | 13:58 |
lucasagomes | in order to know which node it is | 13:58 |
*** trozet has joined #tripleo | 13:58 | |
lucasagomes | devvesa, ok, which driver are u using to deploy? | 13:59 |
devvesa | pxe_drac | 13:59 |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-puppet-elements: Add puppet-kmod into image https://review.openstack.org/276249 | 13:59 |
devvesa | It is a DELL environment | 14:00 |
* lucasagomes checks | 14:00 | |
*** ayoung has quit IRC | 14:01 | |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Increase size of connection tracking table https://review.openstack.org/275769 | 14:01 |
lucasagomes | devvesa, https://github.com/openstack/ironic/blob/stable/liberty/ironic/drivers/drac.py#L46-L53 | 14:02 |
lucasagomes | devvesa, seems like the drac driver is not exposing the lookup method | 14:02 |
lucasagomes | looks like a bug in the driver driver | 14:02 |
lucasagomes | devvesa, can you change that code (1 sec I will give you the snippet), restart the ironic conductor and the node booting? | 14:03 |
devvesa | sure | 14:03 |
lucasagomes | devvesa, ok, on that mapping dict, just add a new element | 14:04 |
lucasagomes | devvesa, 'lookup': self.iscsi_vendor, | 14:04 |
lucasagomes | restart the ironic conductor and the node being deployed | 14:04 |
*** barra204 has quit IRC | 14:04 | |
lucasagomes | devvesa, I'm filing a bug upstream against the drac driver and I if it works we can fix it there and backport to liberty | 14:05 |
devvesa | lucasagomes: Well, since I'm bothering you the whole day, I'm happy to help in the stability of ironic :) | 14:05 |
*** shakamunyi has joined #tripleo | 14:06 | |
*** fgimenez has quit IRC | 14:06 | |
lucasagomes | devvesa, thanks! | 14:06 |
lucasagomes | devvesa, yeah, let's make sure that's the problem | 14:06 |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-puppet-elements: Add puppet-kmod into image https://review.openstack.org/276249 | 14:06 |
*** julim has joined #tripleo | 14:07 | |
*** shadower74 has joined #tripleo | 14:09 | |
*** shadower74 has quit IRC | 14:09 | |
*** fgimenez has joined #tripleo | 14:10 | |
*** fgimenez has joined #tripleo | 14:10 | |
*** jcoufal has joined #tripleo | 14:10 | |
*** jcoufal has quit IRC | 14:10 | |
*** jcoufal has joined #tripleo | 14:11 | |
lucasagomes | devvesa, https://review.openstack.org/#/c/276256/ | 14:15 |
lucasagomes | if that works I will propose a backport one for liberty | 14:15 |
devvesa | lucasagomes: +1 | 14:15 |
lucasagomes | devvesa, oh actually that may be wrong, 1 sec | 14:16 |
lucasagomes | devvesa, cause that's a driver vendor passthru... | 14:16 |
devvesa | ups, ok | 14:16 |
lucasagomes | devvesa, ok patch-set #2 should do it https://review.openstack.org/#/c/276256/2 | 14:17 |
*** pino|work has joined #tripleo | 14:17 | |
*** paramite is now known as paramite|afk | 14:18 | |
*** paramite|afk is now known as paramite | 14:20 | |
*** tiswanso has joined #tripleo | 14:21 | |
*** dshulyak has quit IRC | 14:21 | |
* devvesa is testing | 14:22 | |
*** akuznetsov has quit IRC | 14:24 | |
*** egafford has joined #tripleo | 14:25 | |
*** lblanchard has joined #tripleo | 14:25 | |
marios | bnemec: https://devconfcz2016.sched.org/ | 14:25 |
*** dshulyak has joined #tripleo | 14:26 | |
*** tiswanso has quit IRC | 14:33 | |
openstackgerrit | James Slagle proposed openstack/python-tripleoclient: Simplify checking for stack complete https://review.openstack.org/275149 | 14:33 |
*** tiswanso has joined #tripleo | 14:34 | |
openstackgerrit | James Slagle proposed openstack/python-tripleoclient: Simplify checking for stack complete https://review.openstack.org/275263 | 14:35 |
lucasagomes | devvesa, no dice? | 14:37 |
devvesa | it failed, but I remember to restart the ironic-api after run the overcloud | 14:37 |
*** gmoro has joined #tripleo | 14:38 | |
devvesa | So I consider it a false fail, trying again | 14:38 |
lucasagomes | ack | 14:40 |
lucasagomes | devvesa, you have to restart openstack-ironic-api and conductor btw | 14:40 |
dtantsur | hey folks, what is stable/liberty condition now? is it worth rechecking? | 14:40 |
lucasagomes | because conductor is the one that loads the driver | 14:40 |
devvesa | First I restarted only the conductor | 14:41 |
devvesa | Yes, it fails | 14:42 |
devvesa | And it leaves the node in 'error' state | 14:43 |
devvesa | I could catch the following in the api logs | 14:43 |
devvesa | http://fpaste.org/318434/54596382/ | 14:43 |
devvesa | lucasagomes: ^ | 14:43 |
lucasagomes | devvesa, that's on the second try? | 14:44 |
* lucasagomes looks | 14:44 | |
devvesa | it was the first one. In the second one it left the node on 'error' state and I stopped thinking that I would find another trace in logs | 14:45 |
devvesa | But I couldn't | 14:45 |
lucasagomes | ok, hmm aparently the drac driver wasn't tested with the IPA ramdisk at all :-/ | 14:46 |
*** ayoung has joined #tripleo | 14:47 | |
devvesa | I found an error inside the wsman message in the ironic-conductor | 14:48 |
trown | dtantsur: last full pass for stable/liberty looks like yesterday, but lots of 'all but ha' passes | 14:50 |
trown | dtantsur: probably worth rechecking | 14:50 |
d0ugal | Quick Poll: When you want to check the status of your TripleO deployment, what vital statistics do you check? | 14:50 |
dtantsur | thanks | 14:50 |
lucasagomes | devvesa, can you paste it? | 14:51 |
devvesa | yes. A second | 14:51 |
devvesa | http://fpaste.org/318445/97521145/ lucasagomes: ^ | 14:52 |
devvesa | well ^ | 14:52 |
devvesa | There is a "Import of system configuration XML file operation completed with errors." | 14:53 |
devvesa | at the end | 14:53 |
lucasagomes | ifarkas, ^ any idea ? | 14:53 |
devvesa | make it sense to try the pxe_ipmitool? | 14:53 |
lucasagomes | devvesa, it does yes | 14:54 |
devvesa | ok. let me try | 14:54 |
ifarkas | checking... | 14:54 |
lucasagomes | devvesa, tho the lock error seems not a persistent one | 14:54 |
*** trown is now known as trown|mtg | 14:55 | |
ifarkas | lucasagomes, devvesa, drac driver handles jobs in 'completed with errors' state, so that shouldn't be an issue | 14:55 |
openstackgerrit | Lucas Alvares Gomes proposed openstack/diskimage-builder: Install IPA in a virtual enviroment https://review.openstack.org/235843 | 14:55 |
devvesa | ifakras: ok, thanks! | 14:56 |
*** Goneri has joined #tripleo | 14:56 | |
*** pradk has quit IRC | 14:56 | |
*** pradk has joined #tripleo | 14:56 | |
*** tosky has quit IRC | 14:57 | |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Source undercloud environment variable from a file https://review.openstack.org/275667 | 14:57 |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Split the deploy script into its own file https://review.openstack.org/275668 | 14:57 |
ifarkas | devvesa, does ironic node-show reveals any error message? | 14:58 |
devvesa | | last_error | Failed to tear down. Error: not all arguments converted during string | | 14:58 |
devvesa | | | formatting | 14:58 |
ifarkas | devvesa, can you find anything related to this "not all arguments converted during" in conductor log? | 15:00 |
devvesa | ifarkas: No sorry. I trimmed the error log for the last pastie :( | 15:04 |
devvesa | I'll try now the pxe_ipmitool driver | 15:04 |
ifarkas | devvesa, ack | 15:04 |
*** morazi has quit IRC | 15:07 | |
*** paramite is now known as paramite|afk | 15:09 | |
dtantsur | could someone please approve a trivial DIB change having 2x +2? https://review.openstack.org/#/c/270267/ | 15:13 |
openstackgerrit | Ben Nemec proposed openstack/tripleo-common: Extend ping test create timeout https://review.openstack.org/276293 | 15:20 |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Increase size of connection tracking table https://review.openstack.org/275769 | 15:24 |
*** paramite|afk is now known as paramite | 15:25 | |
*** fgimenez has quit IRC | 15:28 | |
*** fgimenez has joined #tripleo | 15:28 | |
openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Deploy using network isolation the HA job https://review.openstack.org/273424 | 15:32 |
*** julim has quit IRC | 15:33 | |
*** julim has joined #tripleo | 15:35 | |
*** panda_ has quit IRC | 15:37 | |
*** eggmaster has quit IRC | 15:37 | |
*** panda_ has joined #tripleo | 15:38 | |
openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Deploy using network isolation the HA job https://review.openstack.org/273424 | 15:39 |
*** xinwu has joined #tripleo | 15:42 | |
lucasagomes | folks can I get some eyes on https://review.openstack.org/#/c/235843/ ? This will fix the problem of building a fedora ramdisk with IPA | 15:45 |
lucasagomes | dtantsur, ^ tested, works | 15:45 |
dtantsur | just +1ed | 15:46 |
lucasagomes | ty :-) | 15:46 |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Allow the deployer to pick a predefined IP for VIPs https://review.openstack.org/276316 | 15:46 |
*** oshvartz has quit IRC | 15:49 | |
lucasagomes | devvesa, worked with ipmitool? | 15:50 |
*** jistr|mobi has quit IRC | 15:50 | |
devvesa | lucasagomes | 15:59 |
devvesa | Now I have a worse problem. It happened to me sometimes when I deployed VMs | 16:00 |
devvesa | Somehow, it fails the deployment and tries over and over again with different nodes | 16:00 |
devvesa | it tries one node, the node fails (don't know the reason) and then tries another node | 16:01 |
devvesa | This is the trace on the conductor log: http://fpaste.org/318485/54601729/ | 16:02 |
devvesa | I've always solved by creating a new virtual environment | 16:03 |
devvesa | What means not solving anything at all :) | 16:03 |
devvesa | any idea? | 16:04 |
*** trown|mtg is now known as trown | 16:07 | |
*** morazi has joined #tripleo | 16:09 | |
devvesa | lucasagomes: ^ | 16:09 |
lucasagomes | devvesa, that's the RetryFilter in Nova | 16:12 |
lucasagomes | when an instance fails to deploy it will to deploy on another host (by default it tries 3 times) | 16:12 |
devvesa | Do you know why it started to fail now? | 16:13 |
lucasagomes | devvesa, the error is coming from the ramdisk saying it can't start the iSCSI target (it will expose the local disk to the ironic conductor via iSCSI) | 16:14 |
lucasagomes | e o erro foi "No such file or directory" | 16:15 |
lucasagomes | hmm... these nodes have a local disk right? | 16:15 |
devvesa | for sure | 16:16 |
devvesa | This is an error that happened to me on VMs eventually | 16:16 |
trown | devvesa: liberty? | 16:16 |
devvesa | No, it happened more on master | 16:16 |
devvesa | In liberty is the first time I see it | 16:17 |
devvesa | Do you think that is an error that occurs 'after' the previous error? | 16:17 |
devvesa | I mean, that we have solved the previous one and now we are facing a new one? | 16:17 |
*** liverpooler has quit IRC | 16:18 | |
trown | devvesa: could be multiple issues with similar manifestation | 16:19 |
trown | devvesa: I have not seen the "No such file or directory" error on master | 16:19 |
devvesa | trown: it can be... | 16:20 |
*** aufi has quit IRC | 16:20 | |
trown | I have seen that error on the IPA image in RDO, because tgt is not installed, and LIO support is only in mitaka+ | 16:20 |
*** rpothier has joined #tripleo | 16:21 | |
openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: iptables: add missing rule for ceilometer/ssl https://review.openstack.org/276329 | 16:22 |
*** alop has joined #tripleo | 16:23 | |
*** david-lyle has joined #tripleo | 16:24 | |
*** shardy has joined #tripleo | 16:30 | |
lucasagomes | devvesa, yeah off the top of my head I haven't seem that either... I mean we test IPA in gate with VMs | 16:32 |
lucasagomes | but you know can happen... It may be just that the kernel haven't updated the devices tables yet | 16:32 |
lucasagomes | perhaps we need a explicit "partprobe" | 16:32 |
*** hjensas has quit IRC | 16:33 | |
*** akuznetsov has joined #tripleo | 16:40 | |
dtantsur | devvesa, trown++ it might be a dependency problem in IPA itself | 16:44 |
dtantsur | both times I've seen this error it was the cause | 16:44 |
*** fgimenez has quit IRC | 16:45 | |
*** apetrich has quit IRC | 16:45 | |
jpeeler | shardy: could you do a quick container related review https://review.openstack.org/#/c/265366/ | 16:45 |
trown | devvesa: also, are you doing HA all on a smaller virthost? (32G RAM or less)? | 16:45 |
trown | I have very inconsistent results packing all the VMs needed for HA on this size host | 16:46 |
*** olap has quit IRC | 16:46 | |
*** fgimenez has joined #tripleo | 16:47 | |
trown | fwiw, so does our CI | 16:47 |
*** shivrao has joined #tripleo | 16:48 | |
trown | dtantsur: on a slightly related topic (I saw the double scheduler bug in CI recently), would disabling all Nova filters on the undercloud except for the capabilities filter work for the new advanced matching? | 16:48 |
trown | I think that would keep us from hitting the scheduler bug | 16:49 |
*** shivrao has quit IRC | 16:49 | |
dtantsur | trown, it would work, but I'm not convinced it will solve the scheduler race (if you're referring to it) | 16:50 |
devvesa | trown: no. is a baremetal machine of 128g ram | 16:50 |
*** rhallisey has quit IRC | 16:50 | |
trown | dtantsur: I saw https://review.openstack.org/273154 merged, and it made me think that only those filters that got an Exact variant were affected | 16:51 |
trown | maybe that is a naive assumption | 16:51 |
dtantsur | trown, I don't think we're using Exact filters in TripleO | 16:51 |
dtantsur | or rather, someone told me we don't :) | 16:51 |
trown | dtantsur: definitely not, that patch is not even in current-passed-ci yet | 16:52 |
*** shardy has quit IRC | 16:52 | |
trown | dtantsur: oh... I see, I read that totally wrong | 16:52 |
*** jaosorior has quit IRC | 16:52 | |
trown | dtantsur: so that patch does nothing for the tripleo case | 16:52 |
trown | perfect | 16:52 |
trown | since we seem to be the only ones who hit that bug | 16:53 |
*** morazi has quit IRC | 16:53 | |
dtantsur | trown, only tripleo has such a high probability of hitting it, because only tripleo *usually* deploy all the available nodes | 16:54 |
dtantsur | other use cases assume you still have plenty of free resources | 16:54 |
dtantsur | this case is exceptional for them | 16:54 |
trown | right... I just got hopeful that patch would save us | 16:54 |
trown | but it is totally irrelevant to tripleo case | 16:54 |
dtantsur | trown, another set of patches to nova is expected to solve it, but it's for mitaka | 16:54 |
dtantsur | trown, https://blueprints.launchpad.net/nova/+spec/host-state-level-locking | 16:55 |
*** jcoufal has quit IRC | 16:58 | |
*** jtomasek__ has quit IRC | 16:58 | |
*** mcornea has quit IRC | 17:00 | |
*** bnemec has quit IRC | 17:02 | |
*** jistr has quit IRC | 17:02 | |
*** akrivoka has joined #tripleo | 17:03 | |
*** jdob has quit IRC | 17:06 | |
*** jaosorior_ has joined #tripleo | 17:08 | |
*** gfidente has quit IRC | 17:08 | |
*** jaosorior_ is now known as jaosorior | 17:09 | |
*** dsneddon_ has quit IRC | 17:09 | |
*** yamahata has joined #tripleo | 17:14 | |
*** jistr has joined #tripleo | 17:16 | |
*** fgimenez has quit IRC | 17:17 | |
*** jistr has quit IRC | 17:18 | |
*** dtantsur is now known as dtantsur|afk | 17:18 | |
*** mbound has quit IRC | 17:21 | |
*** akuznetsov has quit IRC | 17:23 | |
*** xinwu has quit IRC | 17:25 | |
*** xinwu has joined #tripleo | 17:26 | |
*** tiswanso has quit IRC | 17:32 | |
*** tiswanso has joined #tripleo | 17:32 | |
*** paramite is now known as paramite|afk | 17:34 | |
*** trown is now known as trown|lunch | 17:35 | |
*** athomas has quit IRC | 17:35 | |
*** paramite|afk is now known as paramite | 17:39 | |
*** lucasagomes is now known as lucas-dinner | 17:44 | |
*** mgould is now known as mgould|afk | 17:44 | |
*** paramite has quit IRC | 17:50 | |
*** shivrao has joined #tripleo | 17:51 | |
*** shivrao_ has joined #tripleo | 17:54 | |
*** jistr|mobi has joined #tripleo | 17:56 | |
*** shivrao has quit IRC | 17:56 | |
*** shivrao_ is now known as shivrao | 17:56 | |
*** ayoung has quit IRC | 17:58 | |
*** akrivoka has quit IRC | 17:59 | |
*** Marga_ has quit IRC | 18:05 | |
*** Marga_ has joined #tripleo | 18:05 | |
*** olap has joined #tripleo | 18:07 | |
*** xinwu has joined #tripleo | 18:08 | |
*** ifarkas has quit IRC | 18:17 | |
*** trown|lunch is now known as trown | 18:25 | |
*** trown is now known as trown|mtg | 18:33 | |
*** Marga_ has quit IRC | 18:38 | |
*** devvesa has quit IRC | 18:39 | |
*** yamahata has quit IRC | 18:40 | |
*** shivrao has quit IRC | 18:41 | |
*** eil397 has joined #tripleo | 18:43 | |
*** bvandenh has quit IRC | 18:50 | |
prometheanfire | anyone able to get one more +2? https://review.openstack.org/#/c/273772/ dib | 18:51 |
*** dshulyak has quit IRC | 18:53 | |
*** dshulyak has joined #tripleo | 18:54 | |
*** dsneddon_ has joined #tripleo | 18:58 | |
*** trown|mtg is now known as trown | 18:58 | |
*** tiswanso has quit IRC | 19:02 | |
*** Marga_ has joined #tripleo | 19:04 | |
*** Marga_ has quit IRC | 19:04 | |
*** Marga_ has joined #tripleo | 19:04 | |
*** tiswanso has joined #tripleo | 19:05 | |
*** tiswanso has quit IRC | 19:05 | |
*** tiswanso has joined #tripleo | 19:05 | |
*** akuznetsov has joined #tripleo | 19:06 | |
*** david-lyle has quit IRC | 19:11 | |
*** ayoung has joined #tripleo | 19:13 | |
*** jprovazn has quit IRC | 19:15 | |
*** penick has joined #tripleo | 19:18 | |
*** jaosorior has quit IRC | 19:19 | |
*** electrofelix has quit IRC | 19:22 | |
*** pblaho has quit IRC | 19:36 | |
*** dsneddon_ is now known as dsneddon_brno | 19:37 | |
*** panda_ has quit IRC | 19:37 | |
*** panda_ has joined #tripleo | 19:38 | |
*** shivrao has joined #tripleo | 19:46 | |
*** trozet has quit IRC | 19:48 | |
*** ayoung has quit IRC | 20:04 | |
*** akuznetsov has quit IRC | 20:26 | |
*** dshulyak has quit IRC | 20:31 | |
*** eil397 has quit IRC | 20:38 | |
*** trown is now known as trown|outttypeww | 20:41 | |
*** SlickN1k has joined #tripleo | 20:42 | |
*** goneri_ has joined #tripleo | 20:43 | |
*** gmoro_ has joined #tripleo | 20:43 | |
*** Goneri has quit IRC | 20:47 | |
*** SlickNik has quit IRC | 20:47 | |
*** crinkle has quit IRC | 20:47 | |
*** gmoro has quit IRC | 20:47 | |
*** SlickN1k is now known as SlickNik | 20:47 | |
*** crinkle has joined #tripleo | 20:49 | |
*** crinkle has quit IRC | 20:49 | |
*** bswartz has joined #tripleo | 20:51 | |
*** bswartz has left #tripleo | 20:51 | |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Increase size of connection tracking table https://review.openstack.org/275769 | 20:54 |
*** lblanchard has quit IRC | 21:04 | |
*** NobodyCa1 has joined #tripleo | 21:07 | |
*** absubram has quit IRC | 21:09 | |
*** NobodyCa1 has quit IRC | 21:14 | |
*** goneri_ has quit IRC | 21:21 | |
*** dsneddon_brno has quit IRC | 21:24 | |
*** rpothier has quit IRC | 21:25 | |
*** dsneddon_brno has joined #tripleo | 21:25 | |
*** weshay has quit IRC | 21:29 | |
*** tiswanso has quit IRC | 21:44 | |
*** pradk has quit IRC | 21:46 | |
*** pradk has joined #tripleo | 21:48 | |
*** julim has quit IRC | 21:55 | |
*** julim has joined #tripleo | 21:55 | |
*** julim has quit IRC | 22:02 | |
*** jayg is now known as jayg|g0n3 | 22:05 | |
*** apetrich has joined #tripleo | 22:06 | |
*** absubram has joined #tripleo | 22:06 | |
*** absubram_ has joined #tripleo | 22:08 | |
*** nico_auv has quit IRC | 22:09 | |
*** absubram has quit IRC | 22:11 | |
*** absubram_ is now known as absubram | 22:11 | |
*** absubram has quit IRC | 22:14 | |
*** davidlenwell has quit IRC | 22:22 | |
*** davidlenwell has joined #tripleo | 22:23 | |
*** ChanServ sets mode: +v davidlenwell | 22:23 | |
*** crinkle has joined #tripleo | 22:26 | |
*** david-lyle has joined #tripleo | 22:48 | |
*** egafford has quit IRC | 22:51 | |
*** david-lyle has quit IRC | 22:52 | |
*** goneri_ has joined #tripleo | 22:57 | |
*** ayoung has joined #tripleo | 23:02 | |
*** ayoung_ has joined #tripleo | 23:03 | |
*** eil397 has joined #tripleo | 23:07 | |
*** yamahata has joined #tripleo | 23:10 | |
*** jistr|mobi has quit IRC | 23:19 | |
*** pradk has quit IRC | 23:28 | |
*** chlong has quit IRC | 23:30 | |
*** panda_ has quit IRC | 23:37 | |
*** panda_ has joined #tripleo | 23:38 | |
openstackgerrit | Jeff Peeler proposed openstack/tripleo-heat-templates: Minor fixes to allow local docker registry usage https://review.openstack.org/265366 | 23:39 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!