| mordred | clarkb: although in the context of other sports ... my team has a VERY good player this year | 00:00 |
|---|---|---|
| mordred | clarkb: +3 | 00:00 |
| clarkb | mordred: ty | 00:00 |
| *** felipemonteiro has quit IRC | 00:01 | |
| mordred | clarkb: if things go the way I hope, I should be in a good position to be especially intolerable next march | 00:01 |
| *** bobh has joined #openstack-infra | 00:01 | |
| clarkb | mordred: this is duke? | 00:01 |
| mordred | clarkb: oh yah | 00:02 |
| clarkb | mordred: portland state apparently almost beat duke | 00:02 |
| *** rcernin has quit IRC | 00:02 | |
| clarkb | its probably a good thing that didn't happen or I would have given you crap for it forever | 00:02 |
| mordred | yup. we've had a nice series of closer games where we haven't started playing until the last 5 minutes or so | 00:02 |
| mriedem | the rams happened by dumping jeff fisher | 00:02 |
| mordred | which is clearly not a strategy to continue with ... | 00:02 |
| *** rcernin has joined #openstack-infra | 00:02 | |
| mriedem | and getting baby face mcgee as defensive coordinator from denver | 00:02 |
| mordred | clarkb: yah. and fairly so | 00:02 |
| mriedem | wade philips | 00:02 |
| mordred | mriedem: anybody named 'baby face' is good at their things | 00:03 |
| mriedem | gangsters, r&b singers, nfl coaches | 00:04 |
| mriedem | i guess you're right | 00:04 |
| *** tosky has quit IRC | 00:05 | |
| *** rcernin has quit IRC | 00:07 | |
| *** rcernin has joined #openstack-infra | 00:07 | |
| *** jascott1 has quit IRC | 00:08 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Fix all fails query https://review.openstack.org/524428 | 00:08 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Create ovb-ha-ipv6 and deprecate fs024 job https://review.openstack.org/523903 | 00:08 |
| *** jascott1 has joined #openstack-infra | 00:08 | |
| *** thorst has joined #openstack-infra | 00:11 | |
| *** Goneri has joined #openstack-infra | 00:12 | |
| *** jascott1 has quit IRC | 00:12 | |
| *** david-lyle has quit IRC | 00:15 | |
| *** thorst has quit IRC | 00:15 | |
| clarkb | http://status.openstack.org/elastic-recheck/data/integrated_gate.html it works \o/ | 00:21 |
| clarkb | thanks everyoen | 00:21 |
| clarkb | mriedem: less than 4% categorization rate :( | 00:22 |
| *** sdague has quit IRC | 00:22 | |
| mriedem | i've got a todo to clean out a bunch of stale queries, | 00:23 |
| clarkb | I think some of that may have been the tox siblings thing? | 00:23 |
| mriedem | but haven't watched categorization rate for a long time just because (1) it was all like random unit test stuff or (2) e-r/logstash things just weren't working, probably while the zuulv3 stuff was going on | 00:23 |
| clarkb | mordred: is http://logs.openstack.org/47/500347/12/gate/openstack-tox-pep8/3dff0a9/job-output.txt.gz#_2017-11-30_15_30_20_953565 the bug you fixed? | 00:23 |
| clarkb | I bet if we add a query for that a good chunk of things would be cateogrized and treated as fixed | 00:24 |
| mriedem | go nuts | 00:24 |
| clarkb | mriedem: ya it wasn't until last week that we really got the pipeline working reliably again | 00:24 |
| clarkb | mriedem: but at this point I think it should be happy and healthy and good for real use again | 00:24 |
| mriedem | ok, that's good to know | 00:24 |
| mriedem | i had given up for awhile on stability | 00:24 |
| fungi | there was a period of time where we were happy just to see jobs run and get logs back ;) | 00:25 |
| clarkb | mriedem: I'll have a patch up shortly that should make the reported data a bit better | 00:28 |
| *** david-lyle has joined #openstack-infra | 00:29 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/elastic-recheck master: Query for bug 1735586 https://review.openstack.org/524430 | 00:29 |
| clarkb | mriedem: there | 00:29 |
| openstack | bug 1735586 in OpenStack-Gate "with_dict expects a dict" [Undecided,New] https://launchpad.net/bugs/1735586 | 00:29 |
| clarkb | mordred: ^ you can probably fill in the necessary details on the launchpad bug and close it out | 00:32 |
| *** felipemonteiro_ has quit IRC | 00:34 | |
| *** david-lyle has quit IRC | 00:34 | |
| *** flwang has quit IRC | 00:34 | |
| mordred | clarkb: yah - that should be fixed at this point | 00:35 |
| mordred | I didn't fix it- it was someone else, but it was fixed like, this morning? | 00:36 |
| *** hongbin has quit IRC | 00:36 | |
| clarkb | ya, mostly e-r'ing it as it removes about 2.2k hits | 00:36 |
| mordred | jeez | 00:36 |
| clarkb | so we'll get data for actual problems we should be fixing | 00:36 |
| clarkb | rather than this noise from solved problem | 00:36 |
| mordred | when we break pep8 jobs, we break a lot of things real quick don't we? | 00:36 |
| clarkb | yes | 00:37 |
| mordred | this is amongst the reasons AJaeger and I have been taking our time with depends-on patches for the build-sphinx patch ... | 00:37 |
| * mordred does not want to rush-fix 1k jobs. again. | 00:37 | |
| *** salv-orlando has joined #openstack-infra | 00:42 | |
| *** salv-orlando has quit IRC | 00:46 | |
| mordred | clarkb: if you're still lurking, https://review.openstack.org/#/c/524401/ is ready | 00:48 |
| clarkb | mordred: is the default there not redundant? | 00:52 |
| *** armax has quit IRC | 00:54 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix wrong epel-release-7* package URL https://review.openstack.org/524341 | 00:55 |
| *** thorst has joined #openstack-infra | 00:57 | |
| clarkb | mordred: also what is setting zuul_work_dir? | 00:58 |
| clarkb | looks like we set it in a bunch of roles to default to what the default is there in the playbook | 00:59 |
| clarkb | but not seeing what would set it for that playbook | 00:59 |
| *** kiennt26 has joined #openstack-infra | 01:01 | |
| *** thorst has quit IRC | 01:02 | |
| *** cuongnv has joined #openstack-infra | 01:04 | |
| *** caphrim007_ has joined #openstack-infra | 01:06 | |
| *** rwsu has joined #openstack-infra | 01:07 | |
| *** caphrim007 has quit IRC | 01:09 | |
| *** caphrim007 has joined #openstack-infra | 01:12 | |
| *** esberglu has quit IRC | 01:13 | |
| *** caphrim007_ has joined #openstack-infra | 01:13 | |
| *** ijw has joined #openstack-infra | 01:14 | |
| *** ijw has quit IRC | 01:15 | |
| *** ijw has joined #openstack-infra | 01:15 | |
| *** caphrim007 has quit IRC | 01:16 | |
| *** Apoorva_ has joined #openstack-infra | 01:17 | |
| *** caphrim007_ has quit IRC | 01:17 | |
| openstackgerrit | Stibbons proposed openstack-dev/pbr master: WIP: support pipfile https://review.openstack.org/524436 | 01:19 |
| *** Apoorva has quit IRC | 01:21 | |
| *** Apoorva_ has quit IRC | 01:22 | |
| openstackgerrit | Stibbons proposed openstack-dev/pbr master: WIP: support pipfile https://review.openstack.org/524436 | 01:22 |
| *** ijw has quit IRC | 01:27 | |
| *** sticker has joined #openstack-infra | 01:29 | |
| openstackgerrit | Stibbons proposed openstack-dev/pbr master: WIP: support pipfile https://review.openstack.org/524436 | 01:29 |
| *** thorst has joined #openstack-infra | 01:30 | |
| *** thorst has quit IRC | 01:35 | |
| *** zhurong has joined #openstack-infra | 01:35 | |
| openstackgerrit | Stibbons proposed openstack-dev/pbr master: WIP: support pipfile https://review.openstack.org/524436 | 01:37 |
| *** liujiong has joined #openstack-infra | 01:40 | |
| openstackgerrit | Stibbons proposed openstack-dev/pbr master: WIP: support pipfile https://review.openstack.org/524436 | 01:40 |
| *** wangqian has joined #openstack-infra | 01:42 | |
| *** Goneri has quit IRC | 01:42 | |
| *** salv-orlando has joined #openstack-infra | 01:43 | |
| *** salv-orlando has quit IRC | 01:48 | |
| wangqian | Is the “/project-config/nodepool/nl01.openstack.org.yaml(nl02.openstack.org.yaml)” under the “project-config” project corresponding to the configuration file of the zuul-v3 version of the “nodepool” project ? | 01:49 |
| *** esberglu has joined #openstack-infra | 01:51 | |
| *** esberglu has quit IRC | 01:55 | |
| fungi | wangqian: yes | 01:56 |
| fungi | for zuul v3, nodepool launchers are distributed and coordinate over zookeeper | 01:57 |
| *** inc0 has quit IRC | 01:58 | |
| *** inc0 has joined #openstack-infra | 01:58 | |
| wangqian | fungi: ok thx | 01:59 |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix /dev/pts mount options handling https://review.openstack.org/522654 | 02:01 |
| *** thorst has joined #openstack-infra | 02:01 | |
| *** thorst has quit IRC | 02:07 | |
| *** gmann_afk is now known as gmann | 02:07 | |
| *** niedbalski_ has joined #openstack-infra | 02:13 | |
| *** dbecker_ has joined #openstack-infra | 02:13 | |
| *** rkukura_ has joined #openstack-infra | 02:13 | |
| *** Keitaro1 has joined #openstack-infra | 02:13 | |
| *** rcernin_ has joined #openstack-infra | 02:14 | |
| *** EmilienM_ has joined #openstack-infra | 02:15 | |
| *** thorst has joined #openstack-infra | 02:15 | |
| *** thorst has quit IRC | 02:16 | |
| *** _ari__ has joined #openstack-infra | 02:17 | |
| *** Anticime1 has joined #openstack-infra | 02:18 | |
| *** liujiong_lj has joined #openstack-infra | 02:19 | |
| *** pbourke_ has joined #openstack-infra | 02:19 | |
| *** zerick_ has joined #openstack-infra | 02:19 | |
| *** annp has joined #openstack-infra | 02:20 | |
| *** cinerama` has joined #openstack-infra | 02:20 | |
| *** nunchuck has quit IRC | 02:21 | |
| *** rcernin has quit IRC | 02:21 | |
| *** dbecker has quit IRC | 02:21 | |
| *** niedbalski has quit IRC | 02:21 | |
| *** Dinesh__Bhor has joined #openstack-infra | 02:21 | |
| *** myoung has joined #openstack-infra | 02:21 | |
| *** mattoliverau_ has joined #openstack-infra | 02:21 | |
| *** liujiong has quit IRC | 02:22 | |
| *** zhurong has quit IRC | 02:22 | |
| *** pbourke has quit IRC | 02:22 | |
| *** mhayden has quit IRC | 02:22 | |
| *** eharney has quit IRC | 02:22 | |
| *** jrist has quit IRC | 02:22 | |
| *** markvoelker has quit IRC | 02:22 | |
| *** EmilienM has quit IRC | 02:22 | |
| *** rkukura has quit IRC | 02:22 | |
| *** Dinesh_Bhor has quit IRC | 02:22 | |
| *** mattoliverau has quit IRC | 02:22 | |
| *** fabo has quit IRC | 02:22 | |
| *** _ari_ has quit IRC | 02:22 | |
| *** Nil_ has quit IRC | 02:22 | |
| *** Keitaro has quit IRC | 02:22 | |
| *** zerick has quit IRC | 02:22 | |
| *** Anticimex has quit IRC | 02:22 | |
| *** myoung|ruck has quit IRC | 02:22 | |
| *** adarazs has quit IRC | 02:22 | |
| *** zigo has quit IRC | 02:22 | |
| *** cinerama has quit IRC | 02:22 | |
| *** anupn has quit IRC | 02:22 | |
| *** michaelxin has quit IRC | 02:22 | |
| *** EmilienM_ is now known as EmilienM | 02:22 | |
| *** EmilienM has quit IRC | 02:22 | |
| *** EmilienM has joined #openstack-infra | 02:22 | |
| *** rkukura_ is now known as rkukura | 02:22 | |
| *** mattoliverau_ is now known as mattoliverau | 02:23 | |
| *** myoung is now known as myoung|ruck | 02:23 | |
| *** jrist has joined #openstack-infra | 02:23 | |
| *** patriciadomin has quit IRC | 02:23 | |
| *** bobh has quit IRC | 02:23 | |
| *** dhill_ has quit IRC | 02:23 | |
| *** bandini has quit IRC | 02:23 | |
| *** toabctl has quit IRC | 02:25 | |
| *** bandini has joined #openstack-infra | 02:26 | |
| *** patriciadomin has joined #openstack-infra | 02:26 | |
| *** zigo has joined #openstack-infra | 02:27 | |
| *** anupn has joined #openstack-infra | 02:27 | |
| *** mhayden has joined #openstack-infra | 02:27 | |
| *** markvoelker has joined #openstack-infra | 02:27 | |
| *** fabo has joined #openstack-infra | 02:27 | |
| *** adarazs has joined #openstack-infra | 02:27 | |
| *** Nil_ has joined #openstack-infra | 02:27 | |
| *** toabctl has joined #openstack-infra | 02:27 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Create ovb-ha-ipv6 and deprecate fs024 job https://review.openstack.org/523903 | 02:27 |
| *** zigo is now known as Guest13268 | 02:29 | |
| *** michaelxin has joined #openstack-infra | 02:29 | |
| *** dhinesh has quit IRC | 02:30 | |
| *** dhill_ has joined #openstack-infra | 02:30 | |
| *** nicolasbock has quit IRC | 02:30 | |
| *** dave-mccowan has joined #openstack-infra | 02:30 | |
| *** bandini has quit IRC | 02:30 | |
| *** caphrim007 has joined #openstack-infra | 02:31 | |
| *** bandini has joined #openstack-infra | 02:32 | |
| *** mriedem has quit IRC | 02:33 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Collect /etc/kubernetes/ from CI jobs https://review.openstack.org/523857 | 02:34 |
| *** caphrim007 has quit IRC | 02:35 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add zipl element as s390x architecture bootloader https://review.openstack.org/443548 | 02:35 |
| *** claudiub has quit IRC | 02:40 | |
| *** thorst has joined #openstack-infra | 02:44 | |
| *** salv-orlando has joined #openstack-infra | 02:44 | |
| *** thorst has quit IRC | 02:44 | |
| *** daidv has joined #openstack-infra | 02:45 | |
| *** daidv_ has joined #openstack-infra | 02:45 | |
| EmilienM | infra-root: can someone re-enqueue 524056,1 ? it sounds like it's waiting for nothing and will timeout | 02:48 |
| mordred | clarkb: right - nothing sets it normally in the playbook, which is why it defaults to zuul.project.src_dir, but if a job sets a job variable, the playbook will pick it up like the roles do | 02:48 |
| *** salv-orlando has quit IRC | 02:49 | |
| mordred | clarkb: https://review.openstack.org/#/c/524353/ is an example of doing that | 02:49 |
| *** dave-mccowan has quit IRC | 02:51 | |
| *** Wei_Liu has joined #openstack-infra | 02:56 | |
| *** dhill_ has quit IRC | 02:56 | |
| *** coolsvap has joined #openstack-infra | 02:57 | |
| *** iyamahat_ has quit IRC | 03:08 | |
| *** yamahata has quit IRC | 03:09 | |
| pabelanger | http://paste.openstack.org/show/627935/ | 03:13 |
| pabelanger | EmilienM: ^that is the error I see in ze04.o.o cc mordred | 03:13 |
| pabelanger | I think, zuulv3 might be stuck, but don't want to touch anything until job has a chance to timeout | 03:13 |
| pabelanger | or somebody else looks | 03:14 |
| *** masber has joined #openstack-infra | 03:14 | |
| pabelanger | 2017-12-01 02:49:50.898193 | primary -> localhost | packet_write_wait: Connection to 15.184.66.239 port 22: Broken pipe | 03:15 |
| pabelanger | that is in console log | 03:15 |
| pabelanger | so, it looks like we might have had networking issue in infracloud | 03:15 |
| EmilienM | ok | 03:15 |
| pabelanger | and ansible didn't properly recover | 03:15 |
| EmilienM | so we touch nothing and wait for timeout? | 03:15 |
| pabelanger | yah, zuul should kill the job | 03:15 |
| pabelanger | and if not, then we have a bug | 03:16 |
| pabelanger | but, it is possible the paste above is the reason timeout may not work | 03:16 |
| pabelanger | but, I still see an ansible-playbook process running, so that is good | 03:16 |
| pabelanger | our watchdog should kill it | 03:17 |
| *** ramishra has joined #openstack-infra | 03:18 | |
| *** felipemonteiro has joined #openstack-infra | 03:23 | |
| *** thorst has joined #openstack-infra | 03:25 | |
| *** felipemonteiro_ has joined #openstack-infra | 03:26 | |
| *** felipemonteiro has quit IRC | 03:29 | |
| *** thorst has quit IRC | 03:29 | |
| *** felipemonteiro_ has quit IRC | 03:31 | |
| EmilienM | pabelanger: 524056,1 is queued, what happens? | 03:32 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add Fedora 27 mirror https://review.openstack.org/524456 | 03:32 |
| EmilienM | it sounds like it's stuck again | 03:32 |
| EmilienM | ah it just started | 03:32 |
| EmilienM | nevermind | 03:32 |
| *** rlandy has quit IRC | 03:37 | |
| ianw | pabelanger: is nb03 supposed to be out of rotation? | 03:43 |
| *** salv-orlando has joined #openstack-infra | 03:45 | |
| AJaeger | mordred, config-core: https://review.openstack.org/#/c/524395/1 is ready to go, the new job works fine. Please review this two liner of removing a single broken job. | 03:47 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Remove Fedora 25, add Fedora 27 mirror https://review.openstack.org/524456 | 03:47 |
| *** salv-orlando has quit IRC | 03:49 | |
| *** rcernin has joined #openstack-infra | 03:50 | |
| *** rcernin_ has quit IRC | 03:51 | |
| *** namnh has joined #openstack-infra | 03:52 | |
| openstackgerrit | Merged openstack-infra/project-config master: Change the gate/checks to py3 only for python3 only charms https://review.openstack.org/524182 | 03:54 |
| *** links has joined #openstack-infra | 03:55 | |
| *** udesale has joined #openstack-infra | 03:55 | |
| openstackgerrit | Merged openstack-infra/project-config master: vmware-nsx grafana dashboard https://review.openstack.org/524265 | 04:00 |
| openstackgerrit | Merged openstack-infra/project-config master: fix broken vmware-nsx periodic jobs https://review.openstack.org/524186 | 04:00 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Convert back to zuul.projects https://review.openstack.org/524459 | 04:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Convert back to zuul.projects https://review.openstack.org/524460 | 04:02 |
| *** thorst has joined #openstack-infra | 04:04 | |
| pabelanger | ianw: yah, the plan is to move it back to rackspace, network uploads in vexxhost are just too slow | 04:08 |
| *** nunchuck has joined #openstack-infra | 04:08 | |
| pabelanger | EmilienM: I didn't do anything, so zuul much have done the right thing | 04:08 |
| *** thorst has quit IRC | 04:08 | |
| *** bobh has joined #openstack-infra | 04:09 | |
| ianw | pabelanger: cool ... deliberately stopped and accidentally stopped are hard to distinguish on the server :) | 04:09 |
| pabelanger | ianw: yah, I think there is something in status wiki page too | 04:11 |
| *** hongbin has joined #openstack-infra | 04:12 | |
| *** sree has joined #openstack-infra | 04:19 | |
| dmsimard | clarkb: oh wow, good catch on the missing . yaml | 04:19 |
| *** psachin has joined #openstack-infra | 04:23 | |
| *** threestrands_ has joined #openstack-infra | 04:24 | |
| *** dbecker_ has quit IRC | 04:24 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Convert zuul.projects to a dict https://review.openstack.org/514119 | 04:24 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Remove zuul._projects https://review.openstack.org/524463 | 04:24 |
| *** threestrands has quit IRC | 04:26 | |
| *** daidv has quit IRC | 04:29 | |
| ianw | the field 'args' has an invalid value, which appears to include a variable that is undefined. The error was: 'list object' has no attribute 'values' | 04:34 |
| ianw | oh, bah, ignore me ... of course that's going to fail while "projects" is still a valid list | 04:37 |
| *** dbecker_ has joined #openstack-infra | 04:38 | |
| *** bobh has quit IRC | 04:39 | |
| *** caphrim007 has joined #openstack-infra | 04:40 | |
| *** thorst has joined #openstack-infra | 04:44 | |
| *** pgadiya has joined #openstack-infra | 04:46 | |
| openstackgerrit | Ian Wienand proposed openstack-dev/pbr master: Test on Python 3.6 https://review.openstack.org/524426 | 04:46 |
| *** salv-orlando has joined #openstack-infra | 04:46 | |
| *** bhavik1 has joined #openstack-infra | 04:47 | |
| *** thorst has quit IRC | 04:49 | |
| *** salv-orlando has quit IRC | 04:50 | |
| *** bhavik1 has quit IRC | 04:55 | |
| *** ykarel|away has joined #openstack-infra | 04:55 | |
| *** dhajare has joined #openstack-infra | 04:58 | |
| *** hongbin has quit IRC | 04:59 | |
| *** ykarel|away is now known as ykarel | 05:00 | |
| *** rcernin_ has joined #openstack-infra | 05:09 | |
| *** rcernin has quit IRC | 05:09 | |
| *** david-lyle has joined #openstack-infra | 05:13 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 05:16 | |
| *** janki has joined #openstack-infra | 05:19 | |
| *** janki has quit IRC | 05:20 | |
| *** thorst has joined #openstack-infra | 05:20 | |
| *** janki has joined #openstack-infra | 05:20 | |
| *** esberglu has joined #openstack-infra | 05:23 | |
| *** thorst has quit IRC | 05:25 | |
| *** esberglu has quit IRC | 05:27 | |
| Jeffrey4l | dmsimard, could you review this again https://review.openstack.org/522318 ? | 05:30 |
| EmilienM | pabelanger: good | 05:32 |
| *** gongysh has joined #openstack-infra | 05:32 | |
| *** sticker has quit IRC | 05:36 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Create ovb-ha-ipv6 and deprecate fs024 job https://review.openstack.org/523903 | 05:41 |
| *** gouthamr has quit IRC | 05:44 | |
| *** salv-orlando has joined #openstack-infra | 05:47 | |
| *** salv-orlando has quit IRC | 05:51 | |
| *** thorst has joined #openstack-infra | 05:54 | |
| *** gongysh has quit IRC | 05:56 | |
| *** iyamahat has joined #openstack-infra | 05:57 | |
| *** cshastri has joined #openstack-infra | 05:58 | |
| *** jascott1 has joined #openstack-infra | 05:58 | |
| *** Qiming has quit IRC | 05:59 | |
| *** Qiming has joined #openstack-infra | 06:00 | |
| *** thorst has quit IRC | 06:00 | |
| *** armax has joined #openstack-infra | 06:01 | |
| *** jamesmcarthur has joined #openstack-infra | 06:01 | |
| *** esberglu has joined #openstack-infra | 06:03 | |
| AJaeger | mordred: have a look at swift3 - they pin constraints to an old release and then override ;/ https://review.openstack.org/#/c/488200/ What a mess ;( | 06:04 |
| wangqian | I got some error when I allowed nodepool-launcher，is this below. | 06:04 |
| wangqian | OpenStackCloudHTTPError: (409) Client Error for url: http://172.90.0.2:8774/v2.1/cbd0d9d6a02f4840bc26e00c02f63061/servers Multiple possible networks found, use a Network ID to be more specific. | 06:04 |
| wangqian | I think it need specific a network ,so I write a networks in nodepool.yml,like this below, but it still not work | 06:04 |
| wangqian | pools: | 06:04 |
| wangqian | - name: main | 06:04 |
| wangqian | max-servers: 100 | 06:04 |
| wangqian | labels: | 06:04 |
| wangqian | - name: ubuntu-xenial | 06:04 |
| wangqian | min-ram: 512 | 06:04 |
| wangqian | flavor-name: '1-512-20' | 06:04 |
| wangqian | diskimage: ubuntu-xenial | 06:04 |
| wangqian | key-name: zuul-key | 06:04 |
| AJaeger | wangqian: use paste.openstack.org | 06:04 |
| wangqian | networks: 'e0073eab-7423-4629-a929-16fa25fadd63' | 06:04 |
| *** jamesmcarthur has quit IRC | 06:05 | |
| *** iyamahat has quit IRC | 06:06 | |
| wangqian | http://paste.openstack.org/show/627940/ | 06:06 |
| *** iyamahat has joined #openstack-infra | 06:06 | |
| *** esberglu has quit IRC | 06:08 | |
| timburke | AJaeger: really makes me want to go land https://review.openstack.org/#/c/511964/ and get rid of all of it :-/ | 06:09 |
| *** pcaruana has joined #openstack-infra | 06:10 | |
| *** yamahata has joined #openstack-infra | 06:13 | |
| *** salv-orlando has joined #openstack-infra | 06:14 | |
| AJaeger | timburke: Yeah! | 06:14 |
| AJaeger | timburke: we're removing tox_install.sh - see https://review.openstack.org/#/q/topic:rm-tox_install | 06:14 |
| AJaeger | that continues to use constraints... | 06:15 |
| AJaeger | timburke: 511964 will use the passed in constraints file by default - not the pike one you have | 06:16 |
| *** threestrands_ has quit IRC | 06:24 | |
| *** aeng has quit IRC | 06:26 | |
| *** armax has quit IRC | 06:27 | |
| *** sree_ has joined #openstack-infra | 06:29 | |
| *** thorst has joined #openstack-infra | 06:29 | |
| *** david-lyle has quit IRC | 06:30 | |
| *** sree_ is now known as Guest72141 | 06:30 | |
| tobiash | mordred, jeblair: I've run two nodepools in one tenant, so it is possible but one has to be really careful with that especially with provider naming in nodepool config | 06:32 |
| *** sree has quit IRC | 06:33 | |
| tobiash | if both are named the same (because e.g. the provider has the same name as the cloud it is for) then they cleanup each others nodes without the other noticing... | 06:33 |
| tobiash | and zuul then just responds cannot connect to slave... | 06:33 |
| tobiash | I wouldn't recommend running two production nodepools together in the same tenant | 06:34 |
| tobiash | however the quota should be handled gracefully with the quota patches ;) | 06:34 |
| AJaeger | EmilienM: once they pass, please review https://review.openstack.org/#/q/topic:rm-tox_install+projects:openstack/tripleo | 06:35 |
| tobiash | in production I even don't let nodepool run in the same tenant as the services (because of the very small but deadly risk of a bug in nodepool cleaning up the service vms) | 06:36 |
| tobiash | (in case such a bug would sneak in without being noticed) | 06:36 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove nodesets from builds canceled during reconfiguration https://review.openstack.org/524409 | 06:38 |
| *** thorst has quit IRC | 06:38 | |
| *** dhinesh has joined #openstack-infra | 06:39 | |
| openstackgerrit | Andreas Jaeger proposed openstack/diskimage-builder master: Avoid tox_install.sh for constraints support https://review.openstack.org/524489 | 06:44 |
| *** dhinesh has quit IRC | 06:44 | |
| openstackgerrit | Andreas Jaeger proposed openstack/diskimage-builder master: Avoid tox_install.sh for constraints support https://review.openstack.org/524489 | 06:45 |
| wangqian | I got some error when I run nodepool-launcher,I think it may need specific a network ,so I write a networks in nodepool.yml,but it now work, how to solve this problem ,the message is this blow | 06:58 |
| wangqian | http://paste.openstack.org/show/627941/ | 06:59 |
| wangqian | And the nodepool.yaml is this blow | 07:00 |
| wangqian | http://paste.openstack.org/show/627942/ | 07:00 |
| *** rosmaita has quit IRC | 07:01 | |
| *** tojuvone has joined #openstack-infra | 07:08 | |
| *** tojuvone has left #openstack-infra | 07:08 | |
| *** thorst has joined #openstack-infra | 07:09 | |
| *** jbadiapa has quit IRC | 07:09 | |
| *** thorst has quit IRC | 07:13 | |
| openstackgerrit | Merged openstack-dev/pbr master: Test on Python 3.6 https://review.openstack.org/524426 | 07:18 |
| *** Guest72141 has quit IRC | 07:20 | |
| *** sree has joined #openstack-infra | 07:20 | |
| *** andreas_s has joined #openstack-infra | 07:21 | |
| *** sree has quit IRC | 07:22 | |
| *** sree has joined #openstack-infra | 07:23 | |
| *** e0ne has joined #openstack-infra | 07:27 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/irc-meetings master: Move openstack-rpm-packaging by one hour again https://review.openstack.org/524500 | 07:27 |
| *** rcernin_ has quit IRC | 07:29 | |
| *** ykarel is now known as ykarel|lunch | 07:33 | |
| *** liujiong_lj has quit IRC | 07:34 | |
| *** adisky_ has joined #openstack-infra | 07:43 | |
| openstackgerrit | Rui Chen proposed openstack-infra/nodepool feature/zuulv3: Fix nodepool alien-list issue https://review.openstack.org/522495 | 07:45 |
| *** jtomasek has joined #openstack-infra | 07:47 | |
| *** thorst has joined #openstack-infra | 07:47 | |
| *** jtomasek has quit IRC | 07:47 | |
| *** jtomasek has joined #openstack-infra | 07:47 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Make tripleo-buildimage-overcloud-full-centos-7 voting https://review.openstack.org/524025 | 07:51 |
| *** thorst has quit IRC | 07:52 | |
| *** florianf has joined #openstack-infra | 07:55 | |
| *** kjackal has quit IRC | 08:01 | |
| *** e0ne has quit IRC | 08:03 | |
| *** slaweq has joined #openstack-infra | 08:03 | |
| *** salv-orlando has quit IRC | 08:06 | |
| *** alexchadin has joined #openstack-infra | 08:10 | |
| openstackgerrit | Masayuki Igawa proposed openstack/os-testr master: Remove useless links and indentations https://review.openstack.org/470842 | 08:11 |
| *** yamahata has quit IRC | 08:13 | |
| *** Hal has joined #openstack-infra | 08:18 | |
| *** martinkopec has joined #openstack-infra | 08:21 | |
| AJaeger | frickler, once you're awake and reviewing: https://review.openstack.org/#/c/524395/1 is ready to go, the new job works fine. Could you review this to remove a broken job, please? | 08:23 |
| *** Hal has quit IRC | 08:24 | |
| *** thorst has joined #openstack-infra | 08:25 | |
| *** Hal has joined #openstack-infra | 08:25 | |
| * frickler hands AJaeger a good morning and a +3 ;) | 08:26 | |
| * AJaeger thanks frickler and sends a good morning to him | 08:27 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy-nova-api-ref-src https://review.openstack.org/524395 | 08:29 |
| *** thorst has quit IRC | 08:29 | |
| *** armaan has joined #openstack-infra | 08:32 | |
| *** salv-orlando has joined #openstack-infra | 08:33 | |
| *** migi_ is now known as migi | 08:33 | |
| *** vivek has quit IRC | 08:34 | |
| *** amoralej|off is now known as amoralej | 08:34 | |
| AJaeger | frickler: one more, please: https://review.openstack.org/#/c/524401/ | 08:35 |
| AJaeger | Needed as well for https://review.openstack.org/#/c/524353/ | 08:35 |
| *** kjackal has joined #openstack-infra | 08:35 | |
| *** hashar has joined #openstack-infra | 08:37 | |
| *** ykarel|lunch is now known as ykarel | 08:38 | |
| *** armaan has quit IRC | 08:42 | |
| *** jpena|off is now known as jpena | 08:42 | |
| AJaeger | mordred: quite a few repos using ".[something]" for deps, like oslo.db, oslo.cache ;( | 08:43 |
| *** ralonsoh has joined #openstack-infra | 08:44 | |
| *** esberglu has joined #openstack-infra | 08:48 | |
| *** coolsvap has quit IRC | 08:54 | |
| *** gmann is now known as gmann_afk | 08:54 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 08:56 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Change meeting time and channel https://review.openstack.org/524319 | 08:56 |
| *** thorst has joined #openstack-infra | 08:57 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Change meeting chair for I18n team meeting https://review.openstack.org/524334 | 08:57 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Move openstack-rpm-packaging by one hour again https://review.openstack.org/524500 | 08:57 |
| *** armaan has joined #openstack-infra | 08:57 | |
| *** jpich has joined #openstack-infra | 08:58 | |
| *** jascott1 has quit IRC | 08:59 | |
| *** psachin` has joined #openstack-infra | 08:59 | |
| *** jascott1 has joined #openstack-infra | 08:59 | |
| *** psachin has quit IRC | 09:00 | |
| *** thorst has quit IRC | 09:01 | |
| *** udesale__ has joined #openstack-infra | 09:01 | |
| *** udesale has quit IRC | 09:02 | |
| *** udesale has joined #openstack-infra | 09:04 | |
| *** jascott1 has quit IRC | 09:04 | |
| *** ykarel_ has joined #openstack-infra | 09:04 | |
| *** udesale__ has quit IRC | 09:04 | |
| *** udesale__ has joined #openstack-infra | 09:05 | |
| *** ykarel has quit IRC | 09:06 | |
| *** pgadiya has quit IRC | 09:07 | |
| *** ykarel_ is now known as ykarel|meeting | 09:08 | |
| *** udesale has quit IRC | 09:08 | |
| openstackgerrit | Krzysztof Klimonda proposed openstack-infra/zuul feature/zuulv3: Support autoholding nodes for specific changes/refs https://review.openstack.org/515169 | 09:09 |
| *** iyamahat has quit IRC | 09:14 | |
| *** e0ne has joined #openstack-infra | 09:16 | |
| *** Dinesh__Bhor has quit IRC | 09:18 | |
| jaosorior | ttx: hey, can you check this one out as well https://review.openstack.org/#/c/523124/ ? | 09:18 |
| *** ccamacho has joined #openstack-infra | 09:19 | |
| *** Dinesh__Bhor has joined #openstack-infra | 09:19 | |
| ttx | jaosorior: sure, I can't +2 there though :) | 09:21 |
| jaosorior | ah | 09:22 |
| jaosorior | ttx: thanks anyway though :D | 09:22 |
| *** pgadiya has joined #openstack-infra | 09:23 | |
| *** efoley has joined #openstack-infra | 09:25 | |
| *** thorst has joined #openstack-infra | 09:29 | |
| *** psachin`` has joined #openstack-infra | 09:29 | |
| *** psachin` has quit IRC | 09:30 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/openstack-zuul-jobs master: Add neutron-tempest-plugin to required projects https://review.openstack.org/524540 | 09:32 |
| *** lucas-afk is now known as lucasagomes | 09:33 | |
| *** thorst has quit IRC | 09:34 | |
| *** alexchadin has quit IRC | 09:37 | |
| *** derekh has joined #openstack-infra | 09:39 | |
| *** wangqian has quit IRC | 09:41 | |
| openstackgerrit | Stibbons proposed openstack-dev/pbr master: WIP: support pipfile https://review.openstack.org/524436 | 09:42 |
| *** sree has quit IRC | 09:44 | |
| *** electrofelix has joined #openstack-infra | 09:46 | |
| *** dtantsur|afk is now known as dtantsur | 09:51 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Make api jobs post playbooks honor zuul_work_dir https://review.openstack.org/524401 | 09:51 |
| *** ganso has joined #openstack-infra | 09:51 | |
| *** rossella_s has quit IRC | 09:53 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/project-config master: Remove legacy jobs in nova https://review.openstack.org/514311 | 09:54 |
| frickler | infra-root: gerrit is being pretty slow now over an extended period | 09:54 |
| *** markvoelker has quit IRC | 09:55 | |
| *** jaosorior has quit IRC | 09:55 | |
| *** alexchadin has joined #openstack-infra | 09:55 | |
| *** sree has joined #openstack-infra | 09:58 | |
| *** sree has quit IRC | 09:58 | |
| AJaeger | time for restart again? ;/ | 09:59 |
| *** adisky_ has quit IRC | 10:00 | |
| *** sree has joined #openstack-infra | 10:00 | |
| *** rossella_s has joined #openstack-infra | 10:00 | |
| AJaeger | config-core, a couple of non-urgent cleanup reviews: https://review.openstack.org/524394 https://review.openstack.org/522197 ; and some other smaller reviews: https://review.openstack.org/524262 https://review.openstack.org/522995 https://review.openstack.org/524079 https://review.openstack.org/523924 https://review.openstack.org/512588 | 10:00 |
| *** jamesmcarthur has joined #openstack-infra | 10:01 | |
| *** ykarel_ has joined #openstack-infra | 10:03 | |
| *** cuongnv has quit IRC | 10:04 | |
| *** jamesmcarthur has quit IRC | 10:05 | |
| *** ykarel|meeting has quit IRC | 10:05 | |
| *** daidv_ has quit IRC | 10:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add masakari-dashboard project https://review.openstack.org/516550 | 10:06 |
| *** thorst has joined #openstack-infra | 10:07 | |
| *** alexchadin has quit IRC | 10:07 | |
| *** armaan has quit IRC | 10:10 | |
| *** armaan has joined #openstack-infra | 10:10 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in nova https://review.openstack.org/514310 | 10:11 |
| *** udesale has joined #openstack-infra | 10:11 | |
| *** thorst has quit IRC | 10:11 | |
| *** kiennt26 has quit IRC | 10:12 | |
| *** udesale__ has quit IRC | 10:13 | |
| *** alexchadin has joined #openstack-infra | 10:13 | |
| chandankumar | AJaeger: https://review.openstack.org/#/c/524540/1 for this i think i need to make some changes in playbook to pick the plugins | 10:13 |
| chandankumar | for respective jobs | 10:14 |
| AJaeger | chandankumar: yeah... | 10:14 |
| *** pgadiya has quit IRC | 10:14 | |
| *** claudiub has joined #openstack-infra | 10:14 | |
| AJaeger | chandankumar: or convert them directly in-tree and then to native v3 jobs ;) | 10:14 |
| chandankumar | AJaeger: i will ask neutron guys for the same | 10:15 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Move github webhook from webapp to zuul-web https://review.openstack.org/504267 | 10:15 |
| openstackgerrit | Saad Zaher proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs for freezer https://review.openstack.org/524549 | 10:19 |
| *** namnh has quit IRC | 10:20 | |
| *** tosky has joined #openstack-infra | 10:20 | |
| cmurphy | gerrit kinda sluggish for me today | 10:23 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/openstack-zuul-jobs master: Add neutron-tempest-plugin to dynamic routing playbooks https://review.openstack.org/524540 | 10:27 |
| *** pgadiya has joined #openstack-infra | 10:28 | |
| *** ldnunes has joined #openstack-infra | 10:29 | |
| *** openstackgerrit has quit IRC | 10:33 | |
| *** openstackgerrit has joined #openstack-infra | 10:34 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Serve keys from canonical project name https://review.openstack.org/504807 | 10:34 |
| *** LindaWang1 has joined #openstack-infra | 10:34 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add tripleo-ipsec project https://review.openstack.org/523124 | 10:38 |
| *** gaurangt has quit IRC | 10:39 | |
| *** gaurangt has joined #openstack-infra | 10:40 | |
| *** andreas_s has quit IRC | 10:41 | |
| *** andreas_s has joined #openstack-infra | 10:41 | |
| *** rossella_s has quit IRC | 10:42 | |
| *** jesusaur has quit IRC | 10:42 | |
| *** makowals has quit IRC | 10:42 | |
| *** armaan has quit IRC | 10:44 | |
| *** rossella_s has joined #openstack-infra | 10:44 | |
| *** thorst has joined #openstack-infra | 10:44 | |
| *** armaan has joined #openstack-infra | 10:44 | |
| *** jesusaur has joined #openstack-infra | 10:45 | |
| *** andreas_s has quit IRC | 10:45 | |
| stephenfin | AJaeger: Could you elaborate on what you mean in https://review.openstack.org/#/c/514311/3 ? | 10:46 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config master: Publish governance-sigs content to static.o.o https://review.openstack.org/524557 | 10:46 |
| stephenfin | The only difference I see is that I stuck '-nova' into the titles of tempest-dsvm-lvm and tempest-dsvm-lxc | 10:47 |
| AJaeger | stephenfin: you updated the nova change in the mean time ;) When I gave the -1, you hadn't... | 10:47 |
| * AJaeger double checks | 10:47 | |
| stephenfin | Ah :) | 10:47 |
| stephenfin | AJaeger: I assume I shouldn't move any of the openstack-tox jobs now? | 10:48 |
| *** thorst has quit IRC | 10:48 | |
| AJaeger | stephenfin: the functional ones? You could - but no need. Your call. The PTI mandated ones should stay. | 10:49 |
| *** martinkopec has quit IRC | 10:50 | |
| stephenfin | AJaeger: Aye, those ones. I looked at other projects and could only see them moving "dvsm" tests, but maybe they didn't have functional tests to move | 10:50 |
| *** rossella_s has quit IRC | 10:52 | |
| AJaeger | some have devstack based functional ones - but others use tox. And for those we defined a common job. | 10:52 |
| stephenfin | AJaeger: So it's a case-by-case basis. Fair enough. | 10:52 |
| *** andreas_s has joined #openstack-infra | 10:52 | |
| stephenfin | AJaeger: What do I do about tests that are shared between projects? For example, 'legacy-tempest-dsvm-cells' is run by nova, DevStack and Tempest. Do I move that to nova or leave it where it is? | 10:52 |
| *** sboyron has joined #openstack-infra | 10:53 | |
| *** rossella_s has joined #openstack-infra | 10:53 | |
| AJaeger | stephenfin: you can define jobs in any repo and use them everywhere. So, either place would work ;) | 10:54 |
| AJaeger | The question is more what is this testing and thus where does it belong best to. | 10:54 |
| AJaeger | Might want to talk with andreaf and QA team about that | 10:55 |
| *** markvoelker has joined #openstack-infra | 10:55 | |
| AJaeger | stephenfin: so, if you move it to any repo, please update the job name in project-config for all other users as well. | 10:56 |
| AJaeger | We want only one version.. | 10:56 |
| stephenfin | AJaeger: That makes sense. I assume if nova guys wrote and maintained a test, then the test should belong in that repo | 10:56 |
| *** janki has quit IRC | 10:56 | |
| *** andreas_s has quit IRC | 10:57 | |
| andreaf | stephenfin, AJaeger: core tempest jobs will be hosted in tempest - with core I mean the base ones that can be used by everyone to create their own plus probably those from the integration gate today | 10:58 |
| andreaf | stephenfin, AJaeger: the cells job could easily leave on nova side | 10:58 |
| andreaf | the job must be defined in a repo that runs that job of course :) | 10:59 |
| *** ykarel_ is now known as ykarel | 10:59 | |
| andreaf | and we don't run cells against tempest changes | 10:59 |
| *** kjackal has quit IRC | 11:00 | |
| *** janki has joined #openstack-infra | 11:01 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove Blazar legacy jobs https://review.openstack.org/522197 | 11:01 |
| *** LindaWang1 has quit IRC | 11:02 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Skip releasenotes on driverfixes branches https://review.openstack.org/524262 | 11:03 |
| *** andreas_s has joined #openstack-infra | 11:04 | |
| frickler | AJaeger: question on https://review.openstack.org/#/c/524079/2/zuul.d/projects.yaml : does the "gate: queue: horizon" stanza apply to jobs from templates? or can it be removed when there are no jobs listed for it anymore? | 11:06 |
| *** rossella_s has quit IRC | 11:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update neutron-vpnaas grafana dashboard https://review.openstack.org/522995 | 11:09 |
| *** andreas_s has quit IRC | 11:10 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config master: Move governance tag validation to experimental https://review.openstack.org/524563 | 11:10 |
| *** andreas_s has joined #openstack-infra | 11:10 | |
| AJaeger | frickler: that stanza applies to all horizon jobs and brings the queues together - the co-gating. | 11:10 |
| AJaeger | So, it sohuld stay in there. | 11:11 |
| *** rossella_s has joined #openstack-infra | 11:11 | |
| AJaeger | Does that answer the question or should I explain some more? | 11:11 |
| *** Wei_Liu has quit IRC | 11:12 | |
| frickler | AJaeger: no, thats fine, thx, +2 coming | 11:12 |
| *** thorst has joined #openstack-infra | 11:17 | |
| *** openstackgerrit has quit IRC | 11:18 | |
| *** rossella_s has quit IRC | 11:22 | |
| *** thorst has quit IRC | 11:22 | |
| *** openstackgerrit has joined #openstack-infra | 11:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Drop legacy-horizon-dsvm-tempest-plugin https://review.openstack.org/524079 | 11:25 |
| *** rossella_s has joined #openstack-infra | 11:25 | |
| openstackgerrit | Merged openstack-infra/project-config master: Stop translation job for django-openstack-auth master branch https://review.openstack.org/523924 | 11:25 |
| *** alexchadin has quit IRC | 11:26 | |
| *** alexchadin has joined #openstack-infra | 11:27 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/infra-manual master: zuul v3: Expand section on what *to* convert https://review.openstack.org/524566 | 11:27 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/infra-manual master: Remove version and release https://review.openstack.org/524567 | 11:27 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/infra-manual master: trivial: Remove cruft from 'conf.py' https://review.openstack.org/524568 | 11:27 |
| *** kjackal has joined #openstack-infra | 11:28 | |
| *** udesale has quit IRC | 11:29 | |
| *** Dinesh__Bhor has quit IRC | 11:30 | |
| *** boden has joined #openstack-infra | 11:31 | |
| *** jesusaur has quit IRC | 11:31 | |
| *** jesusaur has joined #openstack-infra | 11:32 | |
| *** wolverineav has joined #openstack-infra | 11:34 | |
| *** alexchadin has quit IRC | 11:34 | |
| *** alexchadin has joined #openstack-infra | 11:34 | |
| stephenfin | AJaeger: Any chance you could take a second look at this today before we merge it? https://review.openstack.org/#/c/522099/ | 11:36 |
| AJaeger | stephenfin: looking | 11:37 |
| AJaeger | stephenfin: zuul.yaml is fine, the rest I expect to be a copy and did not review... | 11:38 |
| stephenfin | AJaeger: Ta. I'll verify that | 11:38 |
| *** Wei_Liu has joined #openstack-infra | 11:40 | |
| *** LindaWang1 has joined #openstack-infra | 11:41 | |
| *** trown has quit IRC | 11:41 | |
| *** LindaWang1 has quit IRC | 11:41 | |
| *** nicolasbock has joined #openstack-infra | 11:43 | |
| *** trown has joined #openstack-infra | 11:44 | |
| *** sree has quit IRC | 11:46 | |
| *** hjensas has joined #openstack-infra | 11:48 | |
| *** esberglu has quit IRC | 11:49 | |
| *** esberglu has joined #openstack-infra | 11:49 | |
| *** salv-orlando has quit IRC | 11:49 | |
| *** salv-orlando has joined #openstack-infra | 11:50 | |
| hjensas | AJaeger: our sphinx job in openstack/networking-baremetal is still failing. (http://logs.openstack.org/35/456235/10/check/build-openstack-sphinx-docs/4ec3da6/job-output.txt.gz#_2017-11-30_19_03_34_216153). You asked us to hold off for topic:update-pti changes a couple of weeks ago. Any eta on when this will be working again? | 11:51 |
| AJaeger | hjensas: Sorry, we run into a few roadblocks to move away. mordred is on it ^ | 11:52 |
| AJaeger | hjensas: let me try something... | 11:53 |
| hjensas | AJaeger: feel free. I was looking at update-pti changes, and this looks related https://review.openstack.org/#/c/521145/ ? | 11:54 |
| *** wolverineav has quit IRC | 11:54 | |
| AJaeger | hjensas: won't work what I tried... | 11:54 |
| *** wolverineav has joined #openstack-infra | 11:55 | |
| *** salv-orlando has quit IRC | 11:55 | |
| AJaeger | hjensas: yes, that will fix your problem... | 11:56 |
| *** thorst has joined #openstack-infra | 11:57 | |
| AJaeger | hjensas: feel free to send a patch for project-config to use publish-openstack-sphinx-docs-neutron instead of publish-openstack-sphinx-docs for networking-baremetal. | 11:57 |
| AJaeger | That only fixes *building* - publishing will still be broken - but allows you to merge content and thus not block you. Once 521145 is in, we can revert that change | 11:58 |
| hjensas | AJaeger: ok, thanks. | 11:59 |
| AJaeger | you'll get my +2 to unblock you - and sorry for the delay here ;( | 11:59 |
| hjensas | sambetts: what do you think ^^ Should we s/-neutron//, or give it some more time? | 12:01 |
| *** thorst has quit IRC | 12:01 | |
| *** sree has joined #openstack-infra | 12:01 | |
| sambetts | hjensas: we need to add "-neutron" not remove it | 12:02 |
| *** mat128 has joined #openstack-infra | 12:02 | |
| hjensas | sambetts: yes, of course my bad. I'll submit the patch. | 12:04 |
| *** mat128 has quit IRC | 12:04 | |
| *** rhallisey has joined #openstack-infra | 12:07 | |
| *** mat128 has joined #openstack-infra | 12:08 | |
| openstackgerrit | Harald Jensås proposed openstack-infra/project-config master: Use publish-openstack-sphinx-docs-neutron https://review.openstack.org/524574 | 12:09 |
| *** alexchadin has quit IRC | 12:10 | |
| *** thorst has joined #openstack-infra | 12:11 | |
| *** rossella_s has quit IRC | 12:11 | |
| *** rossella_s has joined #openstack-infra | 12:12 | |
| *** eharney has joined #openstack-infra | 12:14 | |
| *** lucasagomes is now known as lucas-hungry | 12:14 | |
| *** dhajare has quit IRC | 12:16 | |
| *** armaan has quit IRC | 12:20 | |
| *** sree has quit IRC | 12:21 | |
| AJaeger | config-core, time for a quick unblock review? ^ | 12:22 |
| openstackgerrit | Niraj Singh proposed openstack-infra/project-config master: Add jobs for masakari-dashboard project https://review.openstack.org/516552 | 12:25 |
| *** smatzek has joined #openstack-infra | 12:27 | |
| *** alexchadin has joined #openstack-infra | 12:29 | |
| *** ramishra has quit IRC | 12:32 | |
| *** psachin`` has quit IRC | 12:33 | |
| *** sree has joined #openstack-infra | 12:37 | |
| *** janki has quit IRC | 12:38 | |
| *** rossella_s has quit IRC | 12:39 | |
| *** rossella_s has joined #openstack-infra | 12:40 | |
| *** martinkopec has joined #openstack-infra | 12:43 | |
| *** armaan has joined #openstack-infra | 12:46 | |
| *** sree has quit IRC | 12:47 | |
| *** udesale has joined #openstack-infra | 12:47 | |
| efried | Gerrit is pretty slow this morning. Any known issues? | 12:51 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: Collect /var/lib/docker-puppet. https://review.openstack.org/524576 | 12:52 |
| *** yamamoto has quit IRC | 12:53 | |
| *** yamamoto has joined #openstack-infra | 12:54 | |
| *** rosmaita has joined #openstack-infra | 12:57 | |
| *** jpena is now known as jpena|lunch | 13:00 | |
| *** sree has joined #openstack-infra | 13:04 | |
| *** lucas-hungry is now known as lucasagomes | 13:08 | |
| *** sree has quit IRC | 13:09 | |
| *** yamamoto has quit IRC | 13:09 | |
| *** sree has joined #openstack-infra | 13:12 | |
| *** dtantsur is now known as dtantsur|brb | 13:14 | |
| *** ykarel is now known as ykarel|away | 13:15 | |
| *** sree has quit IRC | 13:17 | |
| *** links has quit IRC | 13:17 | |
| *** raissa has joined #openstack-infra | 13:21 | |
| *** ykarel|away has quit IRC | 13:21 | |
| *** rossella_s has quit IRC | 13:21 | |
| *** efried is now known as fried_rice | 13:22 | |
| *** rossella_s has joined #openstack-infra | 13:22 | |
| flaper87 | fungi: morning! | 13:22 |
| flaper87 | fungi: so, this is not going to become a habit but, I might need to set a hold on another job: ansible-role-k8s-keystone-openshift-centos | 13:22 |
| flaper87 | fungi: different issue but, once again, things seem to work just fine in my dev and not in the gate | 13:23 |
| flaper87 | let me know if it's possible whenever you have time | 13:23 |
| *** yamamoto has joined #openstack-infra | 13:25 | |
| *** markvoelker has quit IRC | 13:25 | |
| *** markvoelker has joined #openstack-infra | 13:25 | |
| *** rossella_s has quit IRC | 13:27 | |
| *** shardy has joined #openstack-infra | 13:28 | |
| *** rossella_s has joined #openstack-infra | 13:30 | |
| *** yamamoto has quit IRC | 13:32 | |
| AJaeger | infra-root, Zuul is not aware of new projects, e.g. masakari-dashboard merged 2:30 h ago and is created at http://git.openstack.org/cgit/openstack/masakari-dashboard - but Zuul does not know it in https://review.openstack.org/#/c/516552 . Similiar for tripleo-ipsec | 13:32 |
| AJaeger | dhellmann: thanks for approving the tox-install removals! | 13:33 |
| *** yamamoto has joined #openstack-infra | 13:33 | |
| dhellmann | AJaeger : there were a few test failures that I didn't look at, but what I did approve should clear a big chunk of the open patches | 13:33 |
| AJaeger | dhellmann: yes, indeed. there are two failures that looked unrelated. | 13:34 |
| *** rlandy has joined #openstack-infra | 13:35 | |
| AJaeger | dhellmann: and then we have quite a few repos using ".[something]" for deps, like oslo.db, oslo.cache. Those need some more work and figuring out how to handle the alternatives... | 13:35 |
| dhellmann | yeah, we need to be able to use "extras" like that to declare optional dependencies | 13:36 |
| dhellmann | we use that feature for drivers, for example | 13:36 |
| AJaeger | mordred: ^ | 13:37 |
| dhellmann | if we have to redo how we declare those it's fine, but we can't lose the feature | 13:37 |
| AJaeger | dhellmann: I expect we need to redo them - but don't know how ;( Any help welcome... | 13:38 |
| * AJaeger was trying to get some repos done to see whether everything works and watch out for problems | 13:38 | |
| dhellmann | there was some talk of having pbr populate extras based on different files, which would let us use -rfilename in tox.ini instead of .[extra] but I don't know how many of those files are supported or how we'd have dynamic ones for things like drivers where the names aren't fixed | 13:39 |
| dhellmann | so docs is an easy one, but postgresql for oslo.db (made up example) would not be something I'd expect pbr to understand | 13:39 |
| *** Wei_Liu has quit IRC | 13:39 | |
| *** pgadiya has quit IRC | 13:40 | |
| *** stephenfin is now known as finucannot | 13:40 | |
| AJaeger | let's see whether tonyb or mordred have some good ideas | 13:41 |
| fried_rice | Gerrit is pretty slow this morning. Any known issues? | 13:43 |
| *** jaypipes is now known as leakypipes | 13:43 | |
| AJaeger | fried_rice: we're waiting for an infra-root to speak some healing words | 13:43 |
| fried_rice | AJaeger Roger that, thanks. | 13:44 |
| pabelanger | just a heads up, moving out of cottage today into actually house :) I'll be offline for a few hours this morning, I hope to return to the interwebs shortly | 13:44 |
| AJaeger | pabelanger: make yourself a proper new home! | 13:45 |
| *** kgiusti has joined #openstack-infra | 13:45 | |
| cmurphy | sooooo sloooooow :( | 13:45 |
| *** dprince has joined #openstack-infra | 13:46 | |
| Shrews | hrm, not seeing the slowness, but i can restart it I suppose | 13:47 |
| fried_rice | Thanks Shrews! | 13:48 |
| *** mriedem has joined #openstack-infra | 13:48 | |
| jlvillal | Shrews, did you restart Gerrit? | 13:48 |
| chandankumar | is https://review.openstack.org/ down? | 13:48 |
| dmsimard | Jeffrey4l: are you still there ? | 13:49 |
| jlvillal | I wish would announce if restarting gerrit. At least here :) | 13:49 |
| AJaeger | Shrews, let's status notice... | 13:49 |
| dmsimard | chandankumar: people have been reporting that it's not very fast | 13:49 |
| *** sree has joined #openstack-infra | 13:49 | |
| fried_rice | I had thought there was a bot that did that. But maybe it's a person each time. | 13:49 |
| chandankumar | dmsimard: np :-) | 13:49 |
| Shrews | jlvillal: yes. should be back now | 13:49 |
| jlvillal | Shrews, Thanks | 13:50 |
| AJaeger | #status notice gerrit has been restarted to get it back to its normal speed. | 13:50 |
| openstackstatus | AJaeger: sending notice | 13:50 |
| Shrews | AJaeger: thx | 13:50 |
| AJaeger | there's the bot ^ ;) | 13:50 |
| rosmaita | speaking of announcing, do we not use #openstack-infra-incident anymore? it still has topic "logs volume is full" from a few months ago | 13:50 |
| AJaeger | infra-root, care to update ^ | 13:50 |
| jlvillal | I had never even heard of that channel | 13:50 |
| dmsimard | rosmaita: it's used for "large scale" incidents | 13:50 |
| dmsimard | to get the noise out of #openstack-infra | 13:51 |
| AJaeger | rosmaita: it's only used a few times a year - for really bad incidents | 13:51 |
| rosmaita | dmsimard: that's fine, but it would be good to clear it out | 13:51 |
| -openstackstatus- NOTICE: gerrit has been restarted to get it back to its normal speed. | 13:51 | |
| AJaeger | rosmaita: agreed | 13:51 |
| *** salv-orlando has joined #openstack-infra | 13:52 | |
| rosmaita | Shrews: thanks for the gerrit restart | 13:53 |
| openstackstatus | AJaeger: finished sending notice | 13:53 |
| *** sree has quit IRC | 13:54 | |
| openstackgerrit | Andreas Jaeger proposed openstack/os-testr master: Avoid tox_install.sh for constraints support https://review.openstack.org/524611 | 13:56 |
| dmsimard | How cool is that, Jeffrey4l has been working on a collectd implementation for kolla's jobs http://logs.openstack.org/50/521450/38/check/kolla-build-centos-source/42ba9e9/rrd_graph/graph.html | 13:57 |
| *** salv-orlando has quit IRC | 13:57 | |
| dmsimard | I wonder if we should scrap my attempt with dstat and use that instead | 13:57 |
| *** yamahata has joined #openstack-infra | 13:58 | |
| *** links has joined #openstack-infra | 13:58 | |
| *** nicolasbock has quit IRC | 13:59 | |
| dmsimard | ah, on a second look there's quite a bit of dependencies required :/ | 13:59 |
| *** bobh has joined #openstack-infra | 14:00 | |
| *** iyamahat has joined #openstack-infra | 14:00 | |
| *** yamamoto has quit IRC | 14:01 | |
| *** jpena|lunch is now known as jpena | 14:01 | |
| mriedem | clarkb: so jobs-output.txt isn't tagged as a console file? | 14:02 |
| *** bnemec has joined #openstack-infra | 14:02 | |
| *** Goneri has joined #openstack-infra | 14:02 | |
| dmsimard | mriedem: it should be? | 14:03 |
| dmsimard | mriedem: http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/submit-logstash-jobs/defaults/main.yaml#n11 | 14:04 |
| mriedem | ah yeah ok it is | 14:05 |
| mriedem | for whatever reason i always have to re-run queries in logstash to get the correct results | 14:05 |
| dmsimard | mriedem: re-run ? like... run twice ? | 14:06 |
| mriedem | yeah | 14:07 |
| mriedem | first result set is always garbage | 14:07 |
| dmsimard | mriedem: that seems symptomatic of an issue I noticed a while back and it was because the elasticsearch cluster wasn't healthy, let me try and see | 14:07 |
| dmsimard | looks up and green, at least according to http://status.openstack.org/elastic-recheck/ | 14:08 |
| dmsimard | mriedem: maybe someone else will know.. it confuses me because sometimes I'll think it's because I'm a kibana noob ;( | 14:09 |
| *** rossella_s has quit IRC | 14:10 | |
| *** esberglu has quit IRC | 14:10 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add a plugin interface for drivers https://review.openstack.org/524620 | 14:10 |
| *** esberglu has joined #openstack-infra | 14:10 | |
| mriedem | dmsimard: what i think happens is kibana tries to 'help' by starting to run the query as soon as you start writing it, like a type of auto-complete, | 14:11 |
| mriedem | but it's not really helpful at all | 14:11 |
| *** bnemec is now known as beekneemech | 14:11 | |
| mriedem | that's purely a guess though | 14:11 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Query for bug 1735586 https://review.openstack.org/524430 | 14:12 |
| openstack | bug 1735586 in OpenStack-Gate "with_dict expects a dict" [Undecided,New] https://launchpad.net/bugs/1735586 | 14:12 |
| *** rossella_s has joined #openstack-infra | 14:12 | |
| flaper87 | fungi: nvm, think I reproduced it | 14:13 |
| *** erlon has joined #openstack-infra | 14:13 | |
| *** amoralej is now known as amoralej|lunch | 14:14 | |
| *** salv-orlando has joined #openstack-infra | 14:15 | |
| *** yamamoto has joined #openstack-infra | 14:17 | |
| *** gibi is now known as giblet | 14:18 | |
| *** ildikov is now known as coffee_cat | 14:20 | |
| *** hashar is now known as hasharAway | 14:22 | |
| *** dansmith is now known as superdan | 14:25 | |
| *** alexchadin has quit IRC | 14:25 | |
| *** cshastri has quit IRC | 14:25 | |
| *** links has quit IRC | 14:26 | |
| *** hasharAway has quit IRC | 14:28 | |
| EmilienM | AJaeger: ack for https://review.openstack.org/#/q/topic:rm-tox_install+projects:openstack/tripleo | 14:30 |
| *** smatzek has quit IRC | 14:31 | |
| *** smatzek has joined #openstack-infra | 14:32 | |
| *** yamamoto has quit IRC | 14:33 | |
| AJaeger | EmilienM: thanks. https://review.openstack.org/#/q/topic:rm-tox_install+is:open contains some related repos as well... | 14:34 |
| EmilienM | AJaeger: ok I'll look as well | 14:34 |
| *** armaan has quit IRC | 14:34 | |
| AJaeger | thanks | 14:35 |
| *** armaan has joined #openstack-infra | 14:35 | |
| *** jkilpatr has quit IRC | 14:35 | |
| *** smatzek has quit IRC | 14:36 | |
| *** panda is now known as panda|lunch | 14:37 | |
| *** martinkopec has quit IRC | 14:39 | |
| mordred | AJaeger: wow - I am very impressed with those patches | 14:39 |
| Jeffrey4l | dmsimard, which dependencies? only require collectd and rrdtool | 14:41 |
| *** slaweq has quit IRC | 14:41 | |
| Jeffrey4l | btw could you review https://review.openstack.org/522318 again. | 14:41 |
| *** slaweq has joined #openstack-infra | 14:41 | |
| *** jamesmcarthur has joined #openstack-infra | 14:42 | |
| AJaeger | mordred: my mini script that helps me http://paste.openstack.org/show/627968/ - so half-automatic (new-branch updates from gerrit and creates a new branch) | 14:43 |
| Jeffrey4l | i hope we move such future to zuul-jobs. | 14:43 |
| dmsimard | Jeffrey4l: epel and centos-opstools repo on centos kind of bothers me | 14:43 |
| Jeffrey4l | hrm. | 14:44 |
| dmsimard | Jeffrey4l: it's fine if you want to use them in kolla but as a base and generic job :/ | 14:44 |
| dmsimard | dstat is built-in in every distro so it has no chance of conflicting with anything | 14:44 |
| AJaeger | mordred: I'm impressed by the speed of approvals so far ;) | 14:44 |
| Jeffrey4l | ok. let me check a bit. | 14:44 |
| Jeffrey4l | and how you dstat works? | 14:44 |
| Jeffrey4l | where i can see the code? | 14:45 |
| AJaeger | mordred: I did not touch neutron/horizon projects - and none that had ".[something]" in requirements | 14:45 |
| mordred | AJaeger: good call | 14:45 |
| mordred | those will take a little more thought | 14:45 |
| AJaeger | bbl | 14:48 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Create ovb-ha-ipv6 and deprecate fs024 job https://review.openstack.org/523903 | 14:53 |
| eumel8 | clarkb, infra-team: please approve https://review.openstack.org/#/c/524400/ so we can start to work with ianw Monday morning | 14:53 |
| jeblair | AJaeger: it looks like puppet-zuul isn't configured to reconfigure zuul on changes to main.yaml. only project-config/zuul/layout/* which is gone now | 14:54 |
| *** d0ugal has quit IRC | 14:56 | |
| *** yamamoto has joined #openstack-infra | 14:59 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/irc-meetings master: Switch release team chair https://review.openstack.org/524636 | 14:59 |
| *** andreas_s has quit IRC | 14:59 | |
| *** smatzek has joined #openstack-infra | 15:00 | |
| *** jascott1 has joined #openstack-infra | 15:00 | |
| *** yamamoto has quit IRC | 15:00 | |
| *** smatzek_ has joined #openstack-infra | 15:01 | |
| *** ramishra has joined #openstack-infra | 15:03 | |
| *** smatzek has quit IRC | 15:04 | |
| *** amoralej|lunch is now known as amoralej | 15:05 | |
| *** hongbin has joined #openstack-infra | 15:05 | |
| *** david-lyle has joined #openstack-infra | 15:07 | |
| *** martinkopec has joined #openstack-infra | 15:11 | |
| *** gouthamr has joined #openstack-infra | 15:11 | |
| *** jkilpatr has joined #openstack-infra | 15:16 | |
| *** caphrim007 has quit IRC | 15:17 | |
| *** caphrim007 has joined #openstack-infra | 15:17 | |
| *** honza has quit IRC | 15:17 | |
| *** d0ugal has joined #openstack-infra | 15:18 | |
| *** jamesmcarthur has quit IRC | 15:19 | |
| *** udesale has quit IRC | 15:20 | |
| *** honza has joined #openstack-infra | 15:20 | |
| *** honza is now known as Guest33545 | 15:20 | |
| *** caphrim007 has quit IRC | 15:22 | |
| *** jamesmcarthur has joined #openstack-infra | 15:22 | |
| *** rbrndt has joined #openstack-infra | 15:22 | |
| *** jcoufal has joined #openstack-infra | 15:22 | |
| *** armaan has quit IRC | 15:24 | |
| *** cshastri has joined #openstack-infra | 15:24 | |
| *** felipemonteiro has joined #openstack-infra | 15:26 | |
| *** dtantsur|brb is now known as dtantsur | 15:27 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:28 | |
| *** ramishra has quit IRC | 15:28 | |
| *** rossella_s has quit IRC | 15:29 | |
| *** d0ugal has quit IRC | 15:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove legacy tips jobs from cliff https://review.openstack.org/524645 | 15:30 |
| dmsimard | I just noticed that some rax nodes have both ipv6 and ipv4 public addresses: http://logs.openstack.org/18/524018/2/check/tripleo-ci-centos-7-containers-multinode/4f135ea/zuul-info/inventory.yaml | 15:30 |
| *** wolverineav has quit IRC | 15:31 | |
| *** armaan has joined #openstack-infra | 15:31 | |
| *** wolverineav has joined #openstack-infra | 15:31 | |
| *** felipemonteiro has quit IRC | 15:31 | |
| *** armax has joined #openstack-infra | 15:32 | |
| *** rossella_s has joined #openstack-infra | 15:32 | |
| *** cshastri has quit IRC | 15:32 | |
| *** camunoz has joined #openstack-infra | 15:33 | |
| dmsimard | configure-unbound is relying on "ansible_default_ipv6" to determine if it should set up ipv6 nameservers: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/roles/configure-unbound/tasks/main.yaml#n12 | 15:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Remove legacy cliff jobs https://review.openstack.org/524646 | 15:33 |
| dmsimard | but when you look at the ansible facts for that host, there's nothing for ansible_default_ipv6: http://logs.openstack.org/18/524018/2/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq-newton/98fa70c/ara/host/ab6aa292-f471-437b-b418-8b4f73927b05/ | 15:33 |
| *** panda|lunch is now known as panda | 15:33 | |
| dmsimard | I'll send a fix to get configure-unbound to address that but I'll also dig to figure out if this is a bug in Ansible | 15:34 |
| dmsimard | If any infra-root could spawn me a node with both ipv4 and ipv6 (or hold me one) I would appreciate it to troubleshoot the issue | 15:34 |
| *** ianw has quit IRC | 15:34 | |
| dmsimard | mwhahaha: ^ fyi | 15:34 |
| mwhahaha | interesting | 15:35 |
| *** yamamoto has joined #openstack-infra | 15:35 | |
| dmsimard | in the multinode setup roles, we have a different approach for detecting ipv6 | 15:35 |
| dmsimard | we rely on the nodepool inventory vars instead | 15:35 |
| *** wolverineav has quit IRC | 15:36 | |
| dmsimard | i.e, http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/multi-node-firewall/tasks/main.yaml#n10 | 15:36 |
| *** martinkopec has quit IRC | 15:36 | |
| dmsimard | mwhahaha: please ping me/us when you see recurring issues like that :D | 15:36 |
| mwhahaha | dmsimard: yea it's just one of those things that happens maybe once a month :D | 15:37 |
| mwhahaha | less so lately | 15:37 |
| mwhahaha | i hadn't seen that one in a while | 15:37 |
| pabelanger | dmsimard: Yah, we depend on shade to default ipv4 / ipv6 in clouds. In some cases, we force ipv4 because glean and ipv6 doesn't work on centos yet | 15:39 |
| *** rfolco has quit IRC | 15:40 | |
| fungi | dmsimard: the log linked above is for a job which ran on a v4-only host | 15:40 |
| fungi | dmsimard: ran in the tripleo test cloud | 15:40 |
| mwhahaha | well it got an ipv6 dns entry so it failed | 15:40 |
| mwhahaha | no | 15:40 |
| *** wolverineav has joined #openstack-infra | 15:40 | |
| pabelanger | dmsimard: https://review.openstack.org/500404/ | 15:41 |
| mwhahaha | i think he linked the wrong log | 15:41 |
| fungi | oh, i was looking at the wrong link you provided | 15:41 |
| dmsimard | fungi: wait, I might have mixed up a few links | 15:41 |
| mwhahaha | http://logs.openstack.org/18/524018/2/check/tripleo-ci-centos-7-containers-multinode/4f135ea/zuul-info/inventory.yaml | 15:41 |
| mwhahaha | http://logs.openstack.org/18/524018/2/check/tripleo-ci-centos-7-containers-multinode/4f135ea/job-output.txt.gz#_2017-11-30_19_32_12_374586 | 15:41 |
| fungi | dmsimard: but yes, all instances in rackspace get ipv4 and ipv6 addresses | 15:41 |
| dmsimard | fungi: ok I mixed up the links indeed -- but the node that ran on RAX still doesn't have an "ansible_default_ipv6" though. | 15:42 |
| dmsimard | http://logs.openstack.org/18/524018/2/check/tripleo-ci-centos-7-containers-multinode/4f135ea/ara/host/341d1a0f-31bd-4ed9-ab49-962a1f9722e2/ | 15:43 |
| dmsimard | There's ipv6 IPs in ansible_all_ipv6_addresses but there's nothing in ansible_default_ipv6 :/ | 15:43 |
| Jeffrey4l | dmsimard, re collectd roles, is the opstools repo not acceptable? epel-release can be removed. | 15:44 |
| fungi | dmsimard: yeah, looking at http://logs.openstack.org/18/524018/2/check/tripleo-ci-centos-7-containers-multinode/4f135ea/zuul-info/zuul-info.primary.txt it seems the v6 interface setup wasn't completed or was somehow undone by the time we ran diagnostics... no flobal v6 address on eth0 and no default v6 route | 15:44 |
| fungi | s/flobal/global/ | 15:44 |
| dmsimard | Jeffrey4l: for things that would be in zuul-jobs we try to keep external dependencies to a minimum since these are meant to be runnable everywhere, even outside of openstack | 15:44 |
| *** thorst has quit IRC | 15:45 | |
| dmsimard | the more packages/mirrors are set up, the higher are the odds of that bleeding into the job and having an impact on results | 15:45 |
| pabelanger | dmsimard: like I said, this is because glean doesn't setup IPv6 on centos, we only support ipv4 | 15:45 |
| pabelanger | so we don't have ipv6 routes | 15:45 |
| dmsimard | pabelanger: so this is a bug only for centos then ? | 15:45 |
| AJaeger | I got some MERGER_FAILURE - anything broken? | 15:46 |
| Jeffrey4l | opstools is the centos official repo. it is live with the centos base/updates repo | 15:46 |
| fungi | dmsimard: the only addresses i see listed in ansible_all_ipv6_addresses are linklocal, so it's no surprise those aren't in ansible_default_ipv6 as they're non-global | 15:46 |
| *** d0ugal has joined #openstack-infra | 15:46 | |
| dmsimard | Jeffrey4l: I know what opstools is :) | 15:46 |
| Jeffrey4l | ah, got. | 15:46 |
| pabelanger | dmsimard: yes, we support ipv6 on ubuntu, and glean should work | 15:46 |
| AJaeger | jeblair: yeah, that would explain it | 15:46 |
| dmsimard | pabelanger: what about other distros ? | 15:46 |
| Jeffrey4l | dmsimard, where is your dstat implement? is it in the zuul already? | 15:46 |
| pabelanger | but we force ipv4 in nodepool, because we cannot mix and match | 15:46 |
| pabelanger | dmsimard: IIRC, just ubuntu support ipv6 in glean | 15:47 |
| dmsimard | Jeffrey4l: I have a quick copy paste from a previous implementation here https://review.openstack.org/#/c/518374/ | 15:47 |
| pabelanger | there is a patch up for gentoo, but never merged | 15:47 |
| dmsimard | Jeffrey4l: need to work on it | 15:47 |
| pabelanger | and opensuse would need to be update too, along with fedora | 15:47 |
| *** kiennt26 has joined #openstack-infra | 15:47 | |
| Jeffrey4l | got. thanks. | 15:47 |
| pabelanger | dmsimard: https://review.openstack.org/367487/ gentoo support | 15:48 |
| dmsimard | pabelanger: ok, for the time being I'll fix configure-unbound -- thanks for letting me know about glean | 15:48 |
| fungi | dmsimard: basically, if you're not familiar with ipv6 networking concepts, addresses in fe80::/10 are not globally routable and aren't supposed to be passed between routers at all (they have to be interface-scoped too since they can conflict on different broadcast domains) | 15:48 |
| *** Guest33545 is now known as honza | 15:48 | |
| pabelanger | dmsimard: a good test would be to confirm it is working on vexxhost, we get ipv6 via dhcp there I believe | 15:48 |
| dmsimard | fungi: makes sense, I just don't recognize those automatically yet :) | 15:48 |
| *** armaan has quit IRC | 15:49 | |
| weshay | hrm.. this is odd.. http://mirror.regionone.infracloud-chocolate.openstack.org/centos/7/ | 15:49 |
| weshay | baseurl=http://mirror.regionone.infracloud-chocolate.openstack.org/centos/centos/$releasever/centosplus/$basearch/ | 15:49 |
| fungi | dmsimard: it's like seeing a 169.254.0.0/16 v4 address on an interface | 15:49 |
| dmsimard | weshay: what is ? | 15:49 |
| weshay | http://logs.openstack.org/94/522294/3/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/360f93f/logs/undercloud/etc/yum.repos.d/CentOS-Base.repo.txt.gz | 15:49 |
| weshay | dmsimard, so I just noticed I can't yum install python-pip | 15:49 |
| *** armaan has joined #openstack-infra | 15:49 | |
| dmsimard | weshay: the double centos ? | 15:49 |
| fungi | it's like twice as good as single centos | 15:50 |
| weshay | oh that too.. but centosplus is not on the mirror | 15:50 |
| weshay | fungi, ha.. totally | 15:50 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Fix double centos in URL for CentOS extras https://review.openstack.org/524653 | 15:50 |
| dmsimard | weshay, fungi: ^ | 15:51 |
| dmsimard | first step | 15:51 |
| *** david-lyle has quit IRC | 15:53 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Synchronize the CentOS Plus mirror https://review.openstack.org/524654 | 15:53 |
| dmsimard | weshay, fungi: ^ second step | 15:53 |
| dmsimard | I just realized that the commit message in 524653 is wrong :/ | 15:54 |
| dmsimard | I wrote extras instead of Plus | 15:54 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Fix double centos in URL for CentOS Plus infrastructure mirror https://review.openstack.org/524653 | 15:55 |
| dmsimard | I hardcore nitted myself ^ jeblair tobiash | 15:55 |
| sc` | if Depends-On changes receive a -1 vote, will the change that depends on them still fire off tests? just trying to figure out if it's a me-problem again | 15:55 |
| dmsimard | sc`: yes | 15:55 |
| dmsimard | sc`: depends-on makes your patch pull that patch and also prevents it from merging until the parent patch merges | 15:56 |
| dmsimard | sc`: there are some exceptions when you're dealing with particular projects such as project-config but zuul will tell you about it as a comment on the review | 15:56 |
| sc` | it doesn't help that i'm doing a chef upgrade with these patches. the gate has to be running the new version before the -1 ones test clean, and i wasn't sure if the meta-patch would be hindered by those dependents | 15:58 |
| openstackgerrit | Eric Fried proposed openstack-infra/os-loganalyze master: WIP: Support ANSI color codes in HTMLView https://review.openstack.org/524658 | 15:58 |
| fried_rice | You've all been waiting for this ^ | 15:59 |
| sc` | the meta-patch, with its own set of modifications to the gate test, has something like a dozen dependent changes | 15:59 |
| fungi | pabelanger: regarding 524654, do you happen to know whether we skipped mirroring centosplus for a reason? | 15:59 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Print a message when we start the Zuul console https://review.openstack.org/524225 | 15:59 |
| fungi | fried_rice: can we get it to also translate ansi blink to the html blink tag? ;) | 15:59 |
| fried_rice | fungi I didn't know there was an ansi blink, but yes. | 16:00 |
| fried_rice | I won't, though. | 16:00 |
| dmsimard | fungi: according to git blame it was never mirrored since infra started mirroring centos at all ~2 years ago | 16:00 |
| fungi | fried_rice: it's sgr 5 (slow blink). there's also a sgr 6 (rapid blink) which was only really supported under ms-dos | 16:01 |
| dmsimard | fungi, pabelanger: FWIW there's not a lot of stuff in centos-plus at all: http://mirror.centos.org/centos/7/centosplus/x86_64/Packages/ | 16:01 |
| fried_rice | fungi Does anyone put blinks in their logs? Tell me no. Please tell me no. | 16:02 |
| dmsimard | weshay: thanks for letting us know about centos plus but python-pip is not in the base OS for centos, it's only in EPEL | 16:02 |
| fungi | fried_rice: i was being facetious about actually adding support for it. i certainly hope nobody ever uses it | 16:03 |
| dmsimard | weshay: alternative is to use python-setuptools (base OS) and then using "easy_install pip" if you really want nothing to do with EPEL | 16:03 |
| AJaeger | Wow, "tripleo-ci-centos-7-containers-multinode TIMED_OUT in 9h 09m 03s" | 16:03 |
| fried_rice | fungi I thought so, but phew. | 16:03 |
| AJaeger | That's a new record ;( | 16:03 |
| dmsimard | AJaeger: that sounds like a bug, there is a hard limit in zuul for timeouts iirc | 16:03 |
| fungi | dmsimard: there is, but it's a timeout per playbook not per job | 16:03 |
| weshay | dmsimard, aye.. thank you. You mentioned that before as well | 16:03 |
| *** Apoorva has joined #openstack-infra | 16:04 | |
| *** slaweq has quit IRC | 16:04 | |
| dmsimard | AJaeger: I'd be curious to see the logs for that run | 16:04 |
| dmsimard | have a link ? | 16:04 |
| *** slaweq has joined #openstack-infra | 16:04 | |
| AJaeger | dmsimard: http://logs.openstack.org/78/524478/2/check/tripleo-ci-centos-7-containers-multinode/0a9948d/ | 16:05 |
| fungi | dmsimard: the ones like that i've seen so far seem to happen when ansible can't reach (or get a response from?) the job node, but rather than giving up quickly it just hangs trying indefinitely until the playbook timeout is reached. then the next playbook kicks off and the same things happens... and the next... | 16:06 |
| *** sree has joined #openstack-infra | 16:06 | |
| dmsimard | yeah we would need executor logs for that one.. http://logs.openstack.org/78/524478/2/check/tripleo-ci-centos-7-containers-multinode/0a9948d/ara/ | 16:06 |
| clarkb | mriedem: I think job output is tagged console but this specific error can only happen in the job output fiel as its part of the job setup not the job itself aiui | 16:07 |
| clarkb | mriedem: so I was being extra specific | 16:07 |
| *** jascott1 has quit IRC | 16:07 | |
| *** Hal has quit IRC | 16:08 | |
| dmsimard | AJaeger, fungi: the timestamps start jumping here: http://logs.openstack.org/78/524478/2/check/tripleo-ci-centos-7-containers-multinode/0a9948d/job-output.txt.gz#_2017-12-01_07_20_34_908823 | 16:08 |
| dmsimard | before that it's okay | 16:08 |
| *** jascott1 has joined #openstack-infra | 16:08 | |
| *** slaweq has quit IRC | 16:09 | |
| *** gagehugo has joined #openstack-infra | 16:09 | |
| pabelanger | fungi: likey to save space in AFS, I think if jobs need it now, we can likey start mirroring it | 16:09 |
| *** udesale has joined #openstack-infra | 16:10 | |
| *** sree has quit IRC | 16:11 | |
| dmsimard | pabelanger: it's a base centos repository and we include it in the centos repository file -- we need to either mirror it or leave the URL to point to the official repos | 16:11 |
| *** jascott1 has quit IRC | 16:12 | |
| *** thorst has joined #openstack-infra | 16:15 | |
| *** sboyron has quit IRC | 16:16 | |
| pabelanger | dmsimard: well, up until now nobody has needed it. So if jobs that weshay pointed out need it, then I don't see a reason not to mirror. | 16:16 |
| pabelanger | dmsimard: otherwise, removing it fine for me too | 16:16 |
| dmsimard | pabelanger: it was a red herring, he needs nothing from there :) | 16:17 |
| mriedem | clarkb: so i might have an idea about why our classification rate is so high for e-r | 16:17 |
| mriedem | clarkb: looking at this trace http://logs.openstack.org/67/518967/2/check/legacy-tempest-dsvm-neutron-dvr/1188c6c/logs/screen-n-cpu.txt.gz?level=TRACE#_Nov_13_19_52_00_826054 | 16:17 |
| mriedem | clarkb: i did a query like: message:"extend_volume" AND message:"VolumePathsNotFound: Could not find any paths for the volume." AND tags:"screen-n-cpu.txt" | 16:17 |
| mriedem | and got 0 results | 16:17 |
| mriedem | when i removed the 'message:"extend_volume"' | 16:17 |
| mriedem | i get 21 hits | 16:17 |
| mriedem | so i'm wondering if something is messed up with the trace parsing/indexing? | 16:17 |
| pabelanger | dmsimard: okay, then I say we fix and leave it disabled | 16:18 |
| clarkb | mriedem: could be multiline parsing problems | 16:19 |
| *** thorst has quit IRC | 16:20 | |
| dmsimard | pabelanger: what does "fix" imply ? Remove it from the repository file and don't synchronize that mirror ? | 16:20 |
| dmsimard | centosplus is mostly about different kernel images, I don't suspect we'll be consuming those | 16:21 |
| dmsimard | so it's really just either we remove it from our repository file or we sync it | 16:21 |
| *** erlon has quit IRC | 16:22 | |
| *** david-lyle has joined #openstack-infra | 16:22 | |
| AJaeger | jeblair: I don't know enough puppet to fix puppet-zuul ;( | 16:23 |
| AJaeger | anybody else around that can help with https://review.openstack.org/523125 - Zuul is not aware of the new repo, it needs a reload. | 16:23 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for volume extend bug 1732199 https://review.openstack.org/524666 | 16:23 |
| openstack | bug 1732199 in OpenStack Compute (nova) "test_extend_attached_volume fails with Unexpected compute_extend_volume result 'Error'" [Medium,Confirmed] https://launchpad.net/bugs/1732199 | 16:23 |
| AJaeger | interesting that this worked in the past. Is puppet run disabled? | 16:23 |
| *** thorst has joined #openstack-infra | 16:24 | |
| AJaeger | infra-root ^ | 16:24 |
| pabelanger | dmsimard: I think you said the URL was wrong, so that would need to be fix, but also, why did the job enable it? That should also be fixed. But, if you want to remove the repo info from configure-mirror, that works too | 16:24 |
| dmsimard | pabelanger: ok let's remove it | 16:25 |
| fungi | AJaeger: i expect the reason it "worked" in the past is that we were updating both the v2 and v3 configuration so the v2 configuration update was triggering the reload | 16:25 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-nova-api-ref-src job https://review.openstack.org/524394 | 16:26 |
| *** holser has joined #openstack-infra | 16:27 | |
| *** armax has quit IRC | 16:27 | |
| AJaeger | fungi: we merged yesterday the governance-sigs repo and that one worked - https://review.openstack.org/#/c/522543/ | 16:27 |
| *** Hal has joined #openstack-infra | 16:28 | |
| AJaeger | wait, that was 24th of November - still without v2 configuration... | 16:28 |
| jeblair | AJaeger: i restarted zuul a couple days ago | 16:28 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Remove the CentOS Plus mirror from the configured mirrors https://review.openstack.org/524653 | 16:28 |
| dmsimard | pabelanger: ^ | 16:28 |
| *** thorst has quit IRC | 16:28 | |
| *** iyamahat has quit IRC | 16:28 | |
| *** yamahata has quit IRC | 16:33 | |
| *** shardy has quit IRC | 16:37 | |
| openstackgerrit | Eric Fried proposed openstack-infra/os-loganalyze master: WIP: Support ANSI color codes in HTMLView https://review.openstack.org/524658 | 16:37 |
| *** dhill_ has joined #openstack-infra | 16:38 | |
| dmsimard | pabelanger, clarkb: are the git farm nodes up to date ? might be worth considering updating as well during the sprint | 16:38 |
| dmsimard | There's been quite a bit of CVEs, including in the kernel | 16:38 |
| pabelanger | dmsimard: we should be running latest centos 7 there | 16:39 |
| pabelanger | but we can confirm | 16:39 |
| *** e0ne has quit IRC | 16:39 | |
| fried_rice | fungi et al FYI http://184.172.12.213/16/524316/1/check/nova-out-of-tree-pvm/f0d1b76/logs/n-cpu.txt.gz is what --^^ does when there are *no* ANSI color codes (I reversed the background and made the fg colors closer to what you see on the console) | 16:39 |
| dmsimard | yeah there's security fixes from 7.3 to 7.4 | 16:39 |
| clarkb | pabelanger: dmsimard its possible we need to do rolling reboots though. We can definitely do those with no downtown if we haproxy correctly | 16:39 |
| fried_rice | We're working up a sample with ANSI codes in it... | 16:39 |
| clarkb | then the frontend is trickier but we can sneak that in on weekend or somethign | 16:40 |
| pabelanger | Description: CentOS Linux release 7.4.1708 (Core) | 16:40 |
| pabelanger | that is on git01.o.o | 16:40 |
| pabelanger | clarkb: ++ | 16:40 |
| dmsimard | ok, cool | 16:40 |
| dmsimard | we don't need to worry about that then | 16:40 |
| *** thorst has joined #openstack-infra | 16:41 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix install-guide publishing https://review.openstack.org/524675 | 16:41 |
| *** rossella_s has quit IRC | 16:42 | |
| fungi | fried_rice: neat! we likely want a toggleable theme for a white-background color scheme and a black-background one. i much prefer the black background like your sample, but i'm probably in the minority on that opinion | 16:42 |
| *** caphrim007 has joined #openstack-infra | 16:42 | |
| *** slaweq has joined #openstack-infra | 16:42 | |
| *** slaweq_ has joined #openstack-infra | 16:42 | |
| *** beekneemech has quit IRC | 16:42 | |
| fried_rice | fungi Toggleable on the client side or via server config? | 16:42 |
| * fried_rice fears you meant the former | 16:43 | |
| fried_rice | ...which is of course harder to do. | 16:43 |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config master: Ensure checkout overrides are correctly set for networking-cisco https://review.openstack.org/512588 | 16:43 |
| *** rossella_s has joined #openstack-infra | 16:44 | |
| weshay | dhellmann, did tripleo set a date on the extension for newton? | 16:44 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Print a message when we start the Zuul console https://review.openstack.org/524225 | 16:45 |
| *** bnemec has joined #openstack-infra | 16:45 | |
| dmsimard | clarkb, pabelanger: could I please get a centos node on rax ? Trying to understand http://logs.openstack.org/18/524018/2/check/tripleo-ci-centos-7-containers-multinode/4f135ea/job-output.txt.gz#_2017-11-30_19_32_12_374586 | 16:45 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for volume extend bug 1732199 https://review.openstack.org/524666 | 16:45 |
| openstack | bug 1732199 in OpenStack Compute (nova) "test_extend_attached_volume fails with Unexpected compute_extend_volume result 'Error'" [Medium,Confirmed] https://launchpad.net/bugs/1732199 | 16:45 |
| clarkb | dmsimard: I think holding nodes is trickier now because its job not cloud/region based. Maybe we should just boot a one off instance? | 16:46 |
| clarkb | pabelanger: ^ am I missing an obvious way to do that better? | 16:46 |
| *** slaweq has quit IRC | 16:46 | |
| dmsimard | Would be convenient to have the roles run on it but I hack it manually | 16:46 |
| dmsimard | s/I hack/I can hack/ | 16:46 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Switch release team chair https://review.openstack.org/524636 | 16:47 |
| AJaeger | infra-root, config-core, I'll be travelling next week. Don't expect any reviews or help from me... | 16:48 |
| dmsimard | AJaeger: you've been doing largely enough | 16:48 |
| fungi | fried_rice: not really sure... if the colors are all set via style properties than it could be as simple as an alternative stylesheet | 16:48 |
| AJaeger | dmsimard: thanks | 16:48 |
| * fungi looks at the page source | 16:48 | |
| clarkb | AJaeger: have a good trip | 16:49 |
| fried_rice | fungi My DHTML is about 15 years old. | 16:49 |
| pabelanger | clarkb: yah, I'm not sure how to hold a running node now. Because zuul has the lock in zookeeper, I think the nodepool hold <id> command will eventually timeout waiting for lock. cc Shrews | 16:49 |
| fungi | fried_rice: yeah, it's all css classes, so swapping stylesheets for that would be pretty trivial | 16:50 |
| clarkb | "do those with no downtown" I feel like my typing drivers are terrible and I need new ones | 16:50 |
| pabelanger | clarkb: maybe we need --force option or soething | 16:50 |
| AJaeger | thanks, clarkb | 16:50 |
| clarkb | pabelanger: I think we want a hold command to hold a running job's nodeset so then we can find a job running on rax and hold that one? | 16:50 |
| *** thorst has quit IRC | 16:50 | |
| fungi | fried_rice: mine too. luckily base sgml and css2 are still relatively effective | 16:50 |
| *** iyamahat has joined #openstack-infra | 16:51 | |
| pabelanger | clarkb: yah, in fact, it is pretty hard today to look at console log and see what node is actually running, that info is only collect and upload to logs.o.o in zuul-info folder. | 16:51 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix branch checkout order https://review.openstack.org/523929 | 16:52 |
| pabelanger | I have to step away again for house things, and food | 16:52 |
| Shrews | pabelanger: clarkb: you can't really force it, not without cooperation from zuul somehow | 16:52 |
| dmsimard | clarkb: I understood the meaning of downtown and just expected that to be a running gag or an expression I was missing out on :p | 16:52 |
| *** andreas_s has joined #openstack-infra | 16:53 | |
| clarkb | dmsimard: unfortunately much less interesting than that, I can't type | 16:53 |
| pabelanger | Shrews: yah, is v2 it was a little easier / nicer to hold a node without zuul knowing. I'd love if we could support that in v3 some how | 16:53 |
| pabelanger | okay, now I run | 16:53 |
| pabelanger | bbiab | 16:53 |
| *** yamamoto has quit IRC | 16:54 | |
| dmsimard | yeah, in v2 you could just do "nodepool hold" on a node even if it was running a job | 16:54 |
| *** rfolco has joined #openstack-infra | 16:55 | |
| *** yamamoto has joined #openstack-infra | 16:56 | |
| jeblair | there's a pending patch to zuul to add more hold features | 16:57 |
| *** claudiub has quit IRC | 16:57 | |
| *** weshay is now known as weshay_mtg | 16:57 | |
| *** andreas_s has quit IRC | 16:57 | |
| jeblair | fried_rice, fungi: you know i love ascii colors, but do we really want to support them in logs? they can be very inconvenient to make sure are working in all the tools we have that work with logs. | 16:59 |
| jeblair | fried_rice, fungi: i spent a lot of time 6 years ago getting them out of devstack runs so we could parse them | 17:00 |
| openstackgerrit | Merged openstack/os-testr master: Fix regex builder https://review.openstack.org/522768 | 17:00 |
| fried_rice | jeblair I was really just wanting to make this work for *my* logs (PowerVM CI). I sorta thought it had a fairly low chance of being adopted os-wide. | 17:00 |
| clarkb | I've just skimmed disk usage for es and logstash workers and everything still looks happy | 17:00 |
| AJaeger | clarkb: then something must be broken ;) | 17:01 |
| fungi | jeblair: for me, about the only reason i can imagine wanting ansi parsing in os-loganalyze is if there is stdout which can't be coerced into dropping color escapes. in which case i'd be slightly in favor of parsing to be able to strip them out rather than parsing to convert them to equivalent html colors (or at least having an option to be able to do that in the browser view) | 17:01 |
| clarkb | AJaeger: ha | 17:01 |
| *** yamamoto has quit IRC | 17:01 | |
| fried_rice | jeblair fungi It annoyed the heck out of me that we had to nix the ANSI color codes from our logs in order to make os-loganalyze display them properly - because then when I download the logs, they're colorless. | 17:01 |
| fried_rice | I personally find colored logs WAY easier to read. | 17:01 |
| fungi | jeblair: granted, i do feel like any console application which emits ansi escapes of any sort when it has no controlling terminal is a bug which should be fixed | 17:02 |
| fried_rice | It would be trivial to write a patch that strips out the ansi color codes in HTMLView entirely, though. So it appears the same as in production today, but retains the codes when I download it. | 17:02 |
| jeblair | fungi, fried_rice: i agree they can be easier to read. but harder to parse and work with. i believe the fact that you had to remove them from your logs is a feature of the current system. | 17:03 |
| jeblair | fried_rice: if anything, it's the *opposite* that should be desired | 17:03 |
| fried_rice | fungi Note that the escapes come about from the log format config. You can take roll your own and get colorless to begin with. | 17:03 |
| *** mat128 has quit IRC | 17:03 | |
| fried_rice | fungi jeblair And even with them included, assuming you're under systemd, journalctl -a will preserve the color codes, but without -a will strip them. | 17:03 |
| jeblair | people (or logstash) should be able to download logs and parse them without having to deal with the escape sequences. display in web pages is harmles... | 17:04 |
| fried_rice | jeblair While I don't disagree with that per se, I also feel people should be able to download logs WITH escape sequences. | 17:04 |
| fungi | i do find the coloring based on log severity in os-loganalyze's rendering to be useful for visual scanning, and worry that random ansi coloring injected throughout would compete with the ability to rely on it | 17:04 |
| fried_rice | Not that it's reasonable to expect CIs to publish two versions. | 17:04 |
| fried_rice | fungi I guess it depends whether a body spends more time looking at logs on the console or in a browser. | 17:05 |
| openstackgerrit | Merged openstack-infra/system-config master: Create mailman server for Kata Containers https://review.openstack.org/524322 | 17:06 |
| fungi | i wonder if a download filter to decorate those logs with ansi escapes based on the same rules os-loganalyze uses to color them with html would make more sense in that case | 17:06 |
| fried_rice | fungi Well, personally I prefer how the ansi color codes color the logs to how os-loganalyze does it in HTMLView. | 17:08 |
| fungi | or add a command-line utility to os-loganalyze to colorize logs based on those same rules | 17:08 |
| fried_rice | IOW, I would rather get the logs in their original format (and be able to have that format include the original ansi escapes) | 17:08 |
| jeblair | fried_rice, fungi: i guess what i'm saying is that we made a choice (a long time ago, when there were fewer of us) to avoid color escape codes in logs due to the complexity it added to systems processing them. we stuck with the lowest common denominator. the lack of support for them in osla and others is a continuous reminder they shouldn't be used and serves to maintain the status quo. if we want to change direction on that, it's ... | 17:08 |
| jeblair | ... worth considering that they may very well start being used and more work may be needed across multiple systems to continue to deal with this. i personally don't want to sign up for that work. if you do, that's great, but please be ready to do more than the minimum if required. :) | 17:08 |
| *** [HeOS] has quit IRC | 17:09 | |
| fried_rice | jeblair Understood. I truly never planned on this getting beyond the PowerVM CI, where I have the power to subvert the norm a little bit :) | 17:09 |
| *** yamahata has joined #openstack-infra | 17:10 | |
| jeblair | fried_rice: experiments always escape from the lab ;) | 17:10 |
| fried_rice | I actually only put a review up in os-loganalyze so I could transfer it easily from my dev machine to my log server :) | 17:10 |
| *** rossella_s has quit IRC | 17:10 | |
| *** rossella_s has joined #openstack-infra | 17:11 | |
| fried_rice | But I'm glad I did, and got y'all's reactions (here and from a couple of devs in -nova). It has been very informative. | 17:11 |
| *** armax has joined #openstack-infra | 17:11 | |
| fungi | yeah, i personally prefer my raw logs to be ascii/utf-8 (which precludes random control characters being embedded since they can do all manner of nasty things in a terminal besides just add color). i'm cool with solutions which make colorization a bolt-on feature for people who still want them | 17:11 |
| *** holser has left #openstack-infra | 17:11 | |
| fungi | however, there are already plenty of command-line filters for doing that too, so adding something in the same vein seems a bit like reinventing the wheel | 17:12 |
| *** Hal has quit IRC | 17:14 | |
| *** fried_rice is now known as fried_rolls | 17:14 | |
| *** slaweq_ has quit IRC | 17:15 | |
| *** rossella_s has quit IRC | 17:15 | |
| fungi | as an aside, it would probably not be too hard with a bit of social engineering to propose an innocuous change for review which generates log content exploiting a common terminal emulator ansi processing/control vulnerability and then convince a reviewer to retrieve the resulting job log and take over their terminal | 17:15 |
| *** slaweq has joined #openstack-infra | 17:16 | |
| fungi | though thankfully most of the egregious ones which and be leveraged to do things like run arbitrary shell commands tend to get patched with great haste when disclosed | 17:16 |
| *** vhosakot has joined #openstack-infra | 17:16 | |
| fungi | s/and be/can be/ | 17:17 |
| *** udesale has quit IRC | 17:17 | |
| *** udesale has joined #openstack-infra | 17:18 | |
| fungi | but friends don't let friends cat untrusted files ;) | 17:18 |
| *** rossella_s has joined #openstack-infra | 17:19 | |
| fungi | even something as innocuous as `tail -f /var/log/some_log_which_can_include_user_supplied_data` is potentially dangerous | 17:20 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: WIP Create a tets matrix role https://review.openstack.org/524402 | 17:20 |
| *** slaweq has quit IRC | 17:20 | |
| *** thorst has joined #openstack-infra | 17:21 | |
| jeblair | fungi: and hopefully emacs and vim have all their automagic code execution vulnerabilities patched.... | 17:22 |
| jeblair | basically, it's hopeless | 17:22 |
| fungi | computers, huh? | 17:23 |
| *** lucasagomes is now known as lucas-afk | 17:24 | |
| *** thorst has quit IRC | 17:26 | |
| mriedem | clarkb: heh you were right http://status.openstack.org/elastic-recheck/data/integrated_gate.html | 17:26 |
| *** derekh has quit IRC | 17:28 | |
| *** thorst has joined #openstack-infra | 17:28 | |
| clarkb | mriedem: ya it was a good chunk of the failures | 17:31 |
| *** thorst has quit IRC | 17:33 | |
| *** thorst has joined #openstack-infra | 17:34 | |
| clarkb | this whole talk of colors sent me down the can I make solarized work localy path | 17:34 |
| clarkb | its a lot more work than I expected because connectbot would require updating 258 color values by hand | 17:35 |
| clarkb | and weechat looks terrible with solraized | 17:35 |
| *** florianf has quit IRC | 17:36 | |
| *** kiennt26 has quit IRC | 17:36 | |
| * fungi wonders where you get those extra 2 colors | 17:36 | |
| clarkb | oh wait its a proper 256 color terminal it just allows you to set background and foreground separate from the other 254 | 17:37 |
| clarkb | but then lists all 256 explicitly | 17:37 |
| *** bnemec has quit IRC | 17:41 | |
| *** slaweq has joined #openstack-infra | 17:42 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't shrink windows on reconfiguration https://review.openstack.org/524410 | 17:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't set job var override_checkout if null https://review.openstack.org/524414 | 17:42 |
| clarkb | that said my astigmatism is probably going to force me into figuring out a more eye friendly color scheme at some point soon | 17:42 |
| *** Apoorva has quit IRC | 17:42 | |
| clarkb | I almost want to just run redshift 24/7 | 17:42 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove the CentOS Plus mirror from the configured mirrors https://review.openstack.org/524653 | 17:43 |
| *** jamesmcarthur has quit IRC | 17:46 | |
| *** udesale has quit IRC | 17:46 | |
| *** jamesmcarthur has joined #openstack-infra | 17:47 | |
| *** jpich has quit IRC | 17:51 | |
| dmsimard | infra-root: I briefly mentioned this sometime last week but we definitely have an issue with nested (qemu, not KVM) virtualization on OVH regions, it seems to reproduce reliably especially on CentOS. We've seen it on opensuse as well. | 17:51 |
| fungi | dmsimard: so virtualization on virtualization, but not hardware-accelerated nested virtualization? | 17:52 |
| clarkb | dmsimard: and we are sure its not kvm? because we ahve confirmed kvm problems in ovh when nested | 17:52 |
| clarkb | qemu on the other hand should just work unless qemu is buggy | 17:52 |
| *** slaweq has quit IRC | 17:52 | |
| dmsimard | A devstack example: https://logs.openstack.org/46/523646/1/check/legacy-tempest-dsvm-neutron-full-centos-7/5bf092c/job-output.txt#_2017-11-29_03_02_38_031560 | 17:52 |
| clarkb | the only thing that the cloud should impact is performance of qemu which should impact workloads in the VMs I guess | 17:52 |
| dmsimard | clarkb: absolutely positive there's no kvm involved | 17:53 |
| dmsimard | all are virt_type=qemu and cpu_mode=none | 17:53 |
| *** slaweq has joined #openstack-infra | 17:53 | |
| clarkb | dmsimard: those are nova settings though right? | 17:53 |
| *** dizquierdo has joined #openstack-infra | 17:53 | |
| clarkb | not necesasrily what is happening in libvirt? | 17:53 |
| clarkb | also that link didn't work for me | 17:53 |
| dmsimard | it's http, not https | 17:54 |
| dmsimard | weird copy pasta | 17:54 |
| dmsimard | http://logs.openstack.org/46/523646/1/check/legacy-tempest-dsvm-neutron-full-centos-7/5bf092c/job-output.txt#_2017-11-29_03_02_38_031560 | 17:54 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: WIP Create a tets matrix role https://review.openstack.org/524402 | 17:55 |
| clarkb | http://logs.openstack.org/46/523646/1/check/legacy-tempest-dsvm-neutron-full-centos-7/5bf092c/logs/libvirt/qemu/instance-00000072.txt.gz says qemu-kvm, not sure if thats normal for proper qemu VMs | 17:56 |
| clarkb | mriedem: ^ do you know? | 17:56 |
| dmsimard | This is an occurrence of opensuse: http://logs.openstack.org/23/522423/7/check/legacy-tempest-dsvm-neutron-full-opensuse-423/b6768d7/job-output.txt#_2017-11-27_21_53_13_340319 | 17:57 |
| mriedem | clarkb: i think that's normal | 17:57 |
| dmsimard | and one in Packstack: http://logs.openstack.org/14/516714/1/check/packstack-integration-scenario002-tempest/7ba8d06/job-output.txt.gz#_2017-10-31_17_55_39_845816 | 17:57 |
| *** slaweq has quit IRC | 17:57 | |
| dmsimard | clarkb: the process to launch the VM is called "qemu-kvm" regardless of the hypervisor http://paste.openstack.org/raw/627986/ | 17:58 |
| clarkb | rax doesn't have nested virt iirc so would be good to compare against a job there | 17:58 |
| *** yamamoto has joined #openstack-infra | 17:58 | |
| clarkb | dmsimard: and lack of accel=kvm means no kvm/virt? | 17:59 |
| dmsimard | clarkb: something like that (that paste is off my home lab) | 17:59 |
| dmsimard | don't have a pure qemu installation on hand | 17:59 |
| clarkb | ya I was comparing against the failed job above (where there are other accel flags but not kvm | 17:59 |
| *** dtantsur is now known as dtantsur|afk | 18:00 | |
| *** sambetts is now known as sambetts|afk | 18:00 | |
| dmsimard | Do we have someone from ovh here ? Otherwise I can ping someone I know. | 18:01 |
| fungi | dmsimard: nobody from the ovh ops team hangs out in here, afaik | 18:01 |
| dmsimard | Ok, I know they use custom kernels so perhaps they've seen this issue before | 18:01 |
| dmsimard | I'll send a ping | 18:01 |
| *** iyamahat has quit IRC | 18:01 | |
| fungi | our only official ovh contacts are the engineering director who approved our pro-bono account, and their support tracker system | 18:02 |
| *** david-lyle has quit IRC | 18:02 | |
| clarkb | I actually did get a card from someone interested in debugging infra specific problems | 18:02 |
| clarkb | so if we can collect a bit more info we can send them email | 18:02 |
| fungi | ooh! | 18:02 |
| *** iyamahat has joined #openstack-infra | 18:03 | |
| fungi | that's a nice turn of events | 18:03 |
| dmsimard | I have a Jean-Daniel as a contact | 18:03 |
| jeblair | mmedvede: i left a comment on 523951 -- i think we want the opposite -- master by default, and an option for older versions. | 18:03 |
| clarkb | dmsimard: ya thats our old contact but in sydney was pointed at damien for technical things like this | 18:03 |
| mmedvede | jeblair: I was under impression we did not want to break people | 18:03 |
| fungi | dmsimard: yeah, jd is who approved our account. most of the time we've reached out to him he's had to farm the question out to someone else, and he tends to not be around/available very often either | 18:03 |
| dmsimard | ok | 18:04 |
| *** david-lyle has joined #openstack-infra | 18:04 | |
| *** weshay_mtg is now known as weshay_bbiab | 18:05 | |
| dmsimard | FWIW we haven't been able to find this occur on tripleo jobs | 18:05 |
| dmsimard | I'm not sure what that means | 18:05 |
| dmsimard | logstashfu is hard | 18:05 |
| jeblair | mmedvede: i don't *want* to break people, but setting an option for backwards compat seems like a good compromise. | 18:05 |
| *** Apoorva has joined #openstack-infra | 18:05 | |
| jeblair | mmedvede: we've already asked people to pin nodepool, for instance | 18:06 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: WIP Create a tets matrix role https://review.openstack.org/524402 | 18:06 |
| clarkb | dmsimard: reading the nova logs the instance continues building the entire log file | 18:06 |
| *** sree has joined #openstack-infra | 18:07 | |
| mmedvede | jeblair: I have misunderstood the original intent then, thanks for the review | 18:07 |
| *** Apoorva has quit IRC | 18:07 | |
| clarkb | dmsimard: so its in the BUILDING state for ~45 minutes | 18:07 |
| clarkb | dmsimard: then the test fails | 18:07 |
| *** yamamoto has quit IRC | 18:08 | |
| jeblair | mmedvede: i'm thinking we add a flag for backwards compat, and give folks maybe a week to set it before we merge into master? | 18:08 |
| mmedvede | jeblair: yes. The thing is, there are going to be people who would break, I guarantee it :) | 18:09 |
| mmedvede | not everyone follows ml | 18:09 |
| mmedvede | their fault, yes, but still | 18:09 |
| jeblair | mmedvede: i agree. it's unfortunate, but running software continuously deployed from git without following the development community is a recipe for such things. | 18:10 |
| dmsimard | Need to brb | 18:10 |
| *** trown is now known as trown|lunch | 18:11 | |
| *** sree has quit IRC | 18:11 | |
| *** myoung|ruck is now known as myoung|ruck|lunc | 18:11 | |
| *** myoung|ruck|lunc is now known as myoung|ruck|food | 18:12 | |
| clarkb | dmsimard: the uuid for the failed instance doesn't show up in any of the libvirt qemu instance logs | 18:14 |
| *** Apoorva has joined #openstack-infra | 18:14 | |
| clarkb | dmsimard: nor does it show up in the libvirt log | 18:14 |
| clarkb | dmsimard: I think this may be a failure in nova | 18:15 |
| clarkb | mriedem: ^ fyi | 18:15 |
| jeblair | AJaeger: hrm, i may have been wrong about the zuul reload issue; i'm digging more | 18:16 |
| jeblair | Dec 1 10:48:55 zuulv3 puppet-user[31569]: (/Stage[main]/Zuul::Scheduler/File[/etc/zuul/layout/main.yaml]/content) content changed '{md5}512ac8999b5029dee322add4d3b10228' to '{md5}59959899f4a58798a27e14a463924081' | 18:17 |
| jeblair | Dec 1 10:48:57 zuulv3 puppet-user[31569]: (/Stage[main]/Zuul::Scheduler/Exec[zuul-reload]) Triggered 'refresh' from 1 events | 18:17 |
| jeblair | that should have done it | 18:17 |
| *** rbrndt has quit IRC | 18:17 | |
| *** gyee has joined #openstack-infra | 18:18 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't shrink windows on reconfiguration https://review.openstack.org/524410 | 18:18 |
| *** dbecker_ has quit IRC | 18:20 | |
| *** jpena is now known as jpena|off | 18:21 | |
| *** slaweq has joined #openstack-infra | 18:21 | |
| jeblair | AJaeger: it looks like there's a pidfile mismatch | 18:22 |
| AJaeger | ;( | 18:23 |
| *** armaan has quit IRC | 18:23 | |
| jeblair | AJaeger: oh neat! | 18:24 |
| clarkb | dmsimard: do you have log evidence of this VM ever actually running (pointing the finger at qemu)? I can't find anything | 18:24 |
| jeblair | AJaeger: it's probably because my work to clean process handling up merged -- i hadn't noticed that :) | 18:24 |
| jeblair | AJaeger: that should make this easy to fix :) | 18:24 |
| clarkb | mriedem: does nova log when it asks libvirt to boot an instance? hoping to cross check logs based on timestamps | 18:24 |
| AJaeger | jeblair: thanks for digging into it and glad to hear it's easy ;) | 18:24 |
| clarkb | mriedem: but not seeing anything that looks like the point when nova says " boot the thing" | 18:24 |
| dhellmann | weshay_bbiab : I'm not sure what you mean about tripleo and a newton extension? maybe EmilienM can answer? | 18:25 |
| jeblair | AJaeger: i think we just need to land https://review.openstack.org/521680 | 18:25 |
| *** slaweq has quit IRC | 18:26 | |
| jeblair | clarkb: do you have any idea what's going wrong in the beaker-rspec test there ^ ? | 18:27 |
| EmilienM | dhellmann: I'm missing context but maybe he meant to say TripleO will not EOL newton before some time? | 18:27 |
| clarkb | jeblair: ya I think that was the stuff I fixed over the last few days. It is a new job now that installs our repos and such | 18:28 |
| dhellmann | EmilienM : that seems like a good guess | 18:28 |
| jeblair | clarkb: oh, is that stuff you fixed recently? | 18:28 |
| clarkb | jeblair: recheck should work | 18:28 |
| jeblair | clarkb: ++ thanks | 18:28 |
| *** tosky has quit IRC | 18:28 | |
| jeblair | ya, that's now the beaker-rspec-infra job | 18:28 |
| EmilienM | dhellmann: we were thinking to keep newton branch until we have stable/queens in place. The reason is tripleo team is currently testing FFU and we might use some upstream CI | 18:29 |
| EmilienM | dhellmann: second reason is we still have a good number of backports | 18:29 |
| *** amoralej is now known as amoralej|off | 18:29 | |
| dhellmann | EmilienM : sounds good to me. I see no urgency to close the branch if it's being used. | 18:29 |
| *** dhill_ has quit IRC | 18:30 | |
| clarkb | jeblair: fwiw the centos job is also broken but apssing for some reason | 18:30 |
| clarkb | jeblair: so I intend on fixing that one soon as well. Hopefully today | 18:30 |
| *** electrofelix has quit IRC | 18:31 | |
| *** armaan has joined #openstack-infra | 18:34 | |
| EmilienM | dhellmann: thanks - we also made sure we don't run useless CI jobs on that branch and made some optimizations | 18:36 |
| eumel8 | clarkb: please approve https://review.openstack.org/#/c/524400/ so we can start to work with ianw Monday morning | 18:37 |
| AJaeger | mordred: me has fun backporting all these changes ;( this is a mess ;( | 18:38 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: zuul: ignore functional jobs when patching tox/setup files https://review.openstack.org/524702 | 18:40 |
| *** mkoderer_ has joined #openstack-infra | 18:41 | |
| mordred | AJaeger: oh - golly - because ... yeah. jeez. ... maybe instead we should make a job variant for stable branches that runs the old tox version of this instead? | 18:42 |
| AJaeger | mordred: we're trying to fix the neutron and horizon requirements as well - so, we would need to the right thing for them. | 18:43 |
| *** ykarel|away has joined #openstack-infra | 18:44 | |
| AJaeger | the problem is the post job which is final - so, will fail with required-projects. | 18:44 |
| AJaeger | mordred: but if there's a good way forward, would be greatly appreciated | 18:45 |
| AJaeger | mordred: this is far bigger than expected ;( | 18:45 |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Make infra beaker rspec job for centos 7 nodes https://review.openstack.org/524705 | 18:47 |
| mordred | AJaeger: yah ... so - maybe we should reorder the work a little bit and get the base job changes in first - that'll fix the post job complication | 18:47 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Use infra centos 7 beaker jobs https://review.openstack.org/524706 | 18:48 |
| clarkb | jeblair: ^ thats the centos 7 stuff | 18:48 |
| AJaeger | mordred: if that works... | 18:48 |
| *** myoung|ruck|food is now known as myoung|ruck | 18:49 | |
| clarkb | eumel8: I'll +2 it but not approve so that ianw can disable puppet as necessary to work through the upgrade steps. Does that work? | 18:49 |
| AJaeger | mordred: sorry, end of long week, I'm tired. So, don't trust me or expect a good overview | 18:49 |
| mordred | AJaeger: lemme write up an etherpad ... although I'm going to rebase the sphinx patch first so we can test the cliff patch (might have an actual solution now!) | 18:49 |
| mordred | AJaeger: oh - no worries from my end - thank you so much for all your work on this so far!!! | 18:49 |
| mordred | AJaeger: I agree - it's wound up being overly complex - there's multiple different issues all interrelated | 18:50 |
| *** raissa has quit IRC | 18:51 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Switch build-openstack-sphinx-docs to build-sphinx-docs https://review.openstack.org/521145 | 18:51 |
| *** raissa has joined #openstack-infra | 18:51 | |
| *** jascott1 has joined #openstack-infra | 18:51 | |
| AJaeger | mordred: thank you for driving this! I mainly told you what's broken ;) | 18:51 |
| *** jamesmcarthur has quit IRC | 18:52 | |
| mordred | clarkb, fungi: got a sec to +2 https://review.openstack.org/#/c/524645/ ? the patch to add new replacements is https://review.openstack.org/#/c/524643 | 18:53 |
| *** jamesmcarthur has joined #openstack-infra | 18:53 | |
| eumel8 | clarkb: thx, you can set additional WF-1 to prevent merges during the weekend | 18:56 |
| AJaeger | mordred: indeed with your horizon/neutron hack, we cover the basics. I wonder about repos that have multiple required-reps right now, like networking-bagpipe which needs openstack/networking-bgpvpn as well | 18:56 |
| AJaeger | mordred: your change fails docs building ;/ | 18:57 |
| AJaeger | could I get a +2A on https://review.openstack.org/#/c/524675/ to fix install-guide publishing, please? | 18:59 |
| *** dhinesh has joined #openstack-infra | 18:59 | |
| *** jascott1 has quit IRC | 18:59 | |
| *** jascott1 has joined #openstack-infra | 18:59 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy tips jobs from cliff https://review.openstack.org/524645 | 19:00 |
| mordred | AJaeger: hrm. that ran tox ... | 19:01 |
| *** iyamahat has quit IRC | 19:01 | |
| AJaeger | argh ;( | 19:02 |
| *** jascott1 has quit IRC | 19:02 | |
| *** dhinesh has quit IRC | 19:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Switch build-openstack-sphinx-docs to build-sphinx-docs https://review.openstack.org/521145 | 19:03 |
| mordred | AJaeger: it was a bad rebase on my part | 19:04 |
| *** jascott1 has joined #openstack-infra | 19:04 | |
| *** Apoorva has quit IRC | 19:04 | |
| *** david-lyle has quit IRC | 19:05 | |
| *** iyamahat has joined #openstack-infra | 19:05 | |
| dmsimard | clarkb: ok I'm back | 19:05 |
| dmsimard | clarkb: I have no idea what's the problem with the VMs at OVH, no :/ | 19:05 |
| mriedem | clarkb: yeah... | 19:05 |
| mriedem | sort of | 19:05 |
| mriedem | it dumps the guest xml before it creates it | 19:06 |
| clarkb | mriedem: in compute log? | 19:06 |
| dmsimard | mriedem: we have cases of VMs failing to build in tempest only at OVH | 19:06 |
| *** Apoorva has joined #openstack-infra | 19:07 | |
| *** kjackal has quit IRC | 19:07 | |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Zuul v3: add per-service default files https://review.openstack.org/521680 | 19:07 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix install-guide publishing https://review.openstack.org/524675 | 19:08 |
| mriedem | clarkb: yeah, i can find an example in a sec | 19:08 |
| clarkb | mriedem: I think I found an example but for the instance with trouble it doesn't do that | 19:08 |
| clarkb | dmsimard: ^ so ya I think it less likely to be a qemu issue and probably more likely nova not actually booting the thing | 19:08 |
| dmsimard | mriedem: would those perhaps ring you a bell ? I extracted some examples here: http://paste.openstack.org/raw/627989/ | 19:08 |
| clarkb | mriedem: http://logs.openstack.org/46/523646/1/check/legacy-tempest-dsvm-neutron-full-centos-7/5bf092c/logs/screen-n-cpu.txt.gz?level=DEBUG instance 2f8de011-b218-4e73-b9e3-e7fcf9e9278b for example | 19:09 |
| *** dhinesh has joined #openstack-infra | 19:09 | |
| dmsimard | Worth mentioning I haven't been able to find this happening on Ubuntu | 19:10 |
| *** rossella_s has quit IRC | 19:10 | |
| clarkb | mriedem: that instance doesn't show up in libvirt logs or qemu instance lgos and there is no dumped xml for it in the n cpu log from what I can see | 19:10 |
| *** Keitaro1 is now known as Keitaro | 19:11 | |
| AJaeger | mordred: works! | 19:11 |
| *** dizquierdo has quit IRC | 19:11 | |
| *** rossella_s has joined #openstack-infra | 19:12 | |
| clarkb | mordred: looks like fungi got that change reviewd for you | 19:12 |
| mriedem | clarkb: this is the guest xml http://logs.openstack.org/46/523646/1/check/legacy-tempest-dsvm-neutron-full-centos-7/5bf092c/logs/screen-n-cpu.txt.gz#_Nov_29_02_16_33_826277 | 19:13 |
| mriedem | http://logs.openstack.org/46/523646/1/check/legacy-tempest-dsvm-neutron-full-centos-7/5bf092c/logs/screen-n-cpu.txt.gz#_Nov_29_02_16_34_248535 | 19:13 |
| mriedem | Nov 29 02:16:34.249773 centos-7-ovh-bhs1-0001106830 nova-compute[7229]: INFO nova.compute.manager [None req-f5b10bda-e8cf-44d6-b3c7-9c6fe8da085d service nova] [instance: 57c7ef47-a65d-4041-9e6f-d8b056b0c4ad] Took 0.74 seconds to spawn the instance on the hypervisor. | 19:13 |
| AJaeger | mordred: I rechecked our test changes | 19:13 |
| mriedem | this is the guest http://logs.openstack.org/46/523646/1/check/legacy-tempest-dsvm-neutron-full-centos-7/5bf092c/logs/libvirt/qemu/instance-0000001e.txt.gz | 19:13 |
| mriedem | clarkb: ^ | 19:13 |
| mordred | AJaeger: \o/ | 19:13 |
| mriedem | however, that shows -uuid 57c7ef47-a65d-4041-9e6f-d8b056b0c4ad | 19:14 |
| mriedem | which is...odd | 19:14 |
| clarkb | mriedem: thats a different file | 19:14 |
| *** vhosakot has quit IRC | 19:14 | |
| clarkb | ya | 19:14 |
| clarkb | 2f8de... is supposed to be the uuid and filepath for the image | 19:14 |
| dmsimard | <cpu match="exact"> doesn't sound right ? | 19:14 |
| clarkb | mriedem: how did you determine that was the qemu instance? | 19:15 |
| mriedem | http://logs.openstack.org/46/523646/1/check/legacy-tempest-dsvm-neutron-full-centos-7/5bf092c/logs/screen-n-cpu.txt.gz#_Nov_29_02_16_33_826277 | 19:15 |
| mriedem | but that's apparently a different instance | 19:15 |
| mriedem | same request id | 19:15 |
| *** dhinesh has quit IRC | 19:15 | |
| mriedem | is this request creating multiple instances? | 19:15 |
| dmsimard | It's tempest so there's bound to be several VMs involved | 19:16 |
| mriedem | no, i mean, same server create request | 19:16 |
| mriedem | multiple vms | 19:16 |
| mriedem | hmm, don't see req-f5b10bda-e8cf-44d6-b3c7-9c6fe8da085d in the conductor or scheduler logs | 19:16 |
| *** rbrndt has joined #openstack-infra | 19:16 | |
| mriedem | oh i know why | 19:17 |
| *** Goneri has quit IRC | 19:17 | |
| mriedem | that's not the server create request id | 19:17 |
| mriedem | req-50ab7479-7407-4070-b674-049ca4a71660 is what we want | 19:17 |
| *** tosky has joined #openstack-infra | 19:19 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Create ovb-ha-ipv6 and deprecate fs024 job https://review.openstack.org/523903 | 19:19 |
| dmsimard | starting a pad to write stuff down to try and see if there's a pattern | 19:20 |
| *** david-lyle has joined #openstack-infra | 19:21 | |
| mriedem | something very weird in these n-cpu logs, | 19:21 |
| mriedem | because the req-50ab7479-7407-4070-b674-049ca4a71660 is now showing up for a swap volume request | 19:21 |
| mriedem | i have seen something like this before, | 19:23 |
| *** jamesmcarthur has quit IRC | 19:23 | |
| mriedem | where it seems like we get request id logging leaked through different operations somehow | 19:23 |
| mriedem | https://bugs.launchpad.net/oslo.log/+bug/1718439 | 19:23 |
| openstack | Launchpad bug 1718439 in oslo.log "Apparent lack of locking in conductor logs" [Undecided,New] | 19:23 |
| mriedem | different issue | 19:24 |
| mriedem | dmsimard: clarkb: this is the last thing i see in the n-cpu logs for that instance while it's spawning | 19:25 |
| mriedem | http://logs.openstack.org/46/523646/1/check/legacy-tempest-dsvm-neutron-full-centos-7/5bf092c/logs/screen-n-cpu.txt.gz#_Nov_29_02_16_33_785677 | 19:25 |
| *** jamesmcarthur has joined #openstack-infra | 19:26 | |
| mriedem | and i think i've seen cases in the past where it seems we just hang during the disk/image stuff during spawn | 19:26 |
| mriedem | in this case, it looks like it's doing file injection? | 19:26 |
| clarkb | ya I noticed what looked like file injection too | 19:26 |
| clarkb | could that be buggy? | 19:26 |
| mriedem | sure | 19:26 |
| mriedem | http://logs.openstack.org/46/523646/1/check/legacy-tempest-dsvm-neutron-full-centos-7/5bf092c/job-output.txt.gz#_2017-11-29_03_02_38_035331 | 19:27 |
| mriedem | that's a file injection test | 19:27 |
| clarkb | could also explain why ubuntu isn't affected if its a version of libguestfs or whatever that is problematic | 19:27 |
| mordred | AJaeger: ok. we're much closer ... there's one more weirdness, but it's actually a bug in the job, so that's good | 19:28 |
| mriedem | http://logs.openstack.org/46/523646/1/check/legacy-tempest-dsvm-neutron-full-centos-7/5bf092c/logs/etc/nova/nova-cpu.conf.txt.gz | 19:28 |
| mriedem | inject_partition = -1 | 19:28 |
| mriedem | means "find the root partition with the file system to mount with libguestfs" | 19:28 |
| mriedem | i guess that's what devstack sets up | 19:29 |
| mriedem | i wonder if maybe there is something going on with privsep | 19:30 |
| mriedem | we should try setting guest.debug=True | 19:30 |
| mriedem | *guestfs.debug | 19:30 |
| mriedem | [guestfs]debug=True in nova.conf | 19:30 |
| *** jascott1 has quit IRC | 19:32 | |
| mriedem | for this particular instance, it doesn't have any personality files | 19:32 |
| mriedem | Inject data image=<LocalFileImage:{'path': '/opt/stack/data/nova/instances/2f8de011-b218-4e73-b9e3-e7fcf9e9278b/disk', 'format': 'qcow2'}> key=None net=None metadata={u'hello': u'world'} admin_password=<SANITIZED> files=[] partition=-1 | 19:32 |
| *** jascott1 has joined #openstack-infra | 19:32 | |
| *** efoley has quit IRC | 19:32 | |
| clarkb | so its just trying to set the one metadata key value pair and the admin password and failing? | 19:33 |
| *** jkilpatr has quit IRC | 19:33 | |
| mriedem | s/failing/hanging/ | 19:33 |
| mriedem | http://logs.openstack.org/46/523646/1/check/legacy-tempest-dsvm-neutron-full-centos-7/5bf092c/job-output.txt.gz#_2017-11-29_03_02_38_031560 | 19:33 |
| mriedem | this particular instance is hanging during injection to the disk image | 19:33 |
| mriedem | using libguestfs | 19:33 |
| clarkb | mriedem: is pushing a patch for guestfs debug to devstack something you want to do? | 19:33 |
| mriedem | well hell yes i can do that | 19:34 |
| clarkb | dmsimard: may not happen to tripleo if tripleo doesn't use file injection | 19:34 |
| mriedem | do you guys have a bug for this? | 19:34 |
| dmsimard | mriedem: not afaik | 19:34 |
| clarkb | its dmsimards things, I haven't filed a bug | 19:34 |
| dmsimard | clarkb: I'm not sure if we do or not. Maybe EmilienM or mwhahaha knows | 19:34 |
| dmsimard | ^ Do we have tempest tests that would run file injection when booting the VMs ? | 19:35 |
| EmilienM | no we don't have that | 19:35 |
| dmsimard | I guess we can just check if we run the same scenario | 19:35 |
| *** armaan has quit IRC | 19:35 | |
| EmilienM | what's the scenario / tempest test that runs it? I can verify | 19:35 |
| dmsimard | EmilienM: it seems default in devstack, do we just do that ping test thing 6 | 19:35 |
| dmsimard | EmilienM: http://logs.openstack.org/46/523646/1/check/legacy-tempest-dsvm-neutron-full-centos-7/5bf092c/job-output.txt#_2017-11-29_03_02_38_031560 | 19:36 |
| dmsimard | tempest.api.compute.servers.test_create_server.ServersTestJSON | 19:36 |
| EmilienM | I confirm we don't run tempest.api.compute.servers.test_create_server.Servers in tripleo ci | 19:36 |
| *** dklyle has joined #openstack-infra | 19:36 | |
| mriedem | dmsimard: clarkb: https://review.openstack.org/524710 | 19:36 |
| dmsimard | mriedem: I'm happy to file a bug but I don't know where to file it | 19:36 |
| *** david-lyle has quit IRC | 19:37 | |
| *** jamesmcarthur has quit IRC | 19:37 | |
| dmsimard | nova ? | 19:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Ensure ChangeLog exists in install-if-python https://review.openstack.org/524712 | 19:37 |
| clarkb | mriedem: +2 thanks, will that just go in the compute log? | 19:37 |
| *** jamesmcarthur has joined #openstack-infra | 19:38 | |
| mriedem | clarkb: this is what the help says, "This configures guestfs to debug messages and push them to Openstack | 19:38 |
| mriedem | logging system. When set to True, it traces libguestfs API calls and | 19:38 |
| mriedem | enable verbose debug messages. In order to use the above feature, | 19:38 |
| mriedem | "libguestfs" package must be installed." | 19:38 |
| mriedem | so we'll see :) | 19:38 |
| mriedem | i for one, am excited | 19:39 |
| openstackgerrit | Sam Yaple proposed openstack-infra/bindep master: Fix logic for groups https://review.openstack.org/517105 | 19:39 |
| fungi | dmsimard: how often does it hit that error condition? | 19:39 |
| *** jamesmcarthur has quit IRC | 19:39 | |
| fungi | on every run in ovh? or infrequently/at random? | 19:39 |
| clarkb | dmsimard: ya I think against nova along the lines of "file injection fails when injecting metadata into the image" | 19:39 |
| clarkb | er | 19:39 |
| clarkb | not fails as mriedem corrected me above | 19:39 |
| clarkb | hangs | 19:39 |
| dmsimard | fungi: it pretty reliably reproduces, we have quite a bit of rechecks in https://review.openstack.org/#/c/517644/ | 19:39 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: WIP Create a tets matrix role https://review.openstack.org/524402 | 19:39 |
| dmsimard | fungi: but only ever on OVH nodes | 19:40 |
| fungi | yeah, then i guess you could turn on debug logging in a proposed change and just recheck a bunch | 19:40 |
| *** rfolco has quit IRC | 19:40 | |
| clarkb | dmsimard: it is probably a race or similar and ovh iops/cpus/whatever tickles it | 19:40 |
| dmsimard | so we went out and looked if it occurred elsewhere and we found cases of opensuse and centos devstack hitting it | 19:40 |
| clarkb | if we can get debug logs from libguestfs then we can either go to the cloud or libguestfs or both with thedata and hopefully get it sorted out | 19:42 |
| dirk | dmsimard: what is the issue ? | 19:42 |
| * dirk sees opensuse mentioned in the irc backlog | 19:42 | |
| *** dhinesh has joined #openstack-infra | 19:42 | |
| * AJaeger waves good night | 19:42 | |
| dmsimard | dirk: tempest failing to successfully create VMs on the OVH clouds | 19:42 |
| fungi | night AJaeger, thanks for all the help! | 19:42 |
| mordred | AJaeger: goodnight - have a good weekend and a good week off next week! | 19:42 |
| dmsimard | dirk: not just opensuse, centos too. | 19:42 |
| clarkb | dirk: looks to be due to nova file injection hanging with libguestfs | 19:42 |
| AJaeger | thanks | 19:43 |
| clarkb | might want to compare libguestfs versions across ubuntu and centos and suse | 19:43 |
| *** AJaeger is now known as AJaeger_ | 19:43 | |
| clarkb | maybe ubuntus is older or newer than the other two | 19:43 |
| mriedem | this is the last thing we see before it's gone http://logs.openstack.org/46/523646/1/check/legacy-tempest-dsvm-neutron-full-centos-7/5bf092c/logs/screen-n-cpu.txt.gz#_Nov_29_02_16_33_785830 | 19:43 |
| mriedem | that's when it's importing guestfs | 19:43 |
| mriedem | and does this 'inspect_capabilities' thing | 19:43 |
| dirk | So guests hangs or the host starts to hang? | 19:43 |
| mriedem | i don't even see us get past that to attempt to inject the metadata | 19:43 |
| dirk | We have a pile of patches in guestfs :-/ | 19:44 |
| *** leakypipes has quit IRC | 19:44 | |
| mriedem | dirk: guest | 19:44 |
| mriedem | other instances are getting created ok | 19:44 |
| dmsimard | Filing a bug against nova with what we've found, we can move it elsewhere if it's not an issue in nova later | 19:46 |
| mordred | AJaeger_: I know you're out - but wanted to let you know - the cliff docs job passed!!! | 19:46 |
| mriedem | so i think this is where we hang https://github.com/openstack/nova/blob/b6a245f0425a07be3871a976952646d2bdd44533/nova/virt/disk/vfs/guestfs.py#L75 | 19:47 |
| mriedem | which now that i'm looking at that, | 19:48 |
| mriedem | plus the crazy context switch on the logging with the request id, | 19:48 |
| mriedem | my bets are on eventlet | 19:48 |
| mriedem | clarkb: dmsimard: ^ | 19:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix scheduler reconfiguration handler https://review.openstack.org/524714 | 19:48 |
| mriedem | superdan: maybe you have ideas, but w/o reading all of this traceback, | 19:48 |
| mriedem | superdan: we see guestfs hang here https://github.com/openstack/nova/blob/b6a245f0425a07be3871a976952646d2bdd44533/nova/virt/disk/vfs/guestfs.py#L75 | 19:48 |
| mriedem | and right at the same time, i start seeing the logging in n-cpu switch from one request to another, using the same request id, but definitely a different request (different logs for different instances, same request id) | 19:49 |
| *** kjackal has joined #openstack-infra | 19:49 | |
| mriedem | is there some trick to avoid an eventlet context switch here? | 19:49 |
| superdan | I'm guessing the switch is actually in the tpool line below it yeah? | 19:49 |
| mriedem | yeah | 19:50 |
| mriedem | the thread was added here https://review.openstack.org/#/c/181808/ | 19:50 |
| superdan | you wouldn't want to lock that whole region | 19:50 |
| mriedem | so this probably explains why it appears to hang, | 19:50 |
| mriedem | eventlet switches, | 19:50 |
| mriedem | the guestfs thing ends but we don't get that response so we just never keep going on this thread | 19:50 |
| dirk | mriedem: do we know since when it happens? There was a qemu/libvirt maintenance update recently | 19:51 |
| superdan | does guestfs really finish? | 19:51 |
| superdan | that's a good place to hang forever | 19:51 |
| mriedem | idk if it does, we have a devstack patch up to enable guestfs debug | 19:51 |
| mriedem | dirk: there are no errors in the logs so no | 19:52 |
| dmsimard | We have had this issue recorded as far back as the logs allow us to go, oct 31st | 19:52 |
| dirk | Ok, so not the recent update, good | 19:52 |
| SamYaple | clarkb: https://review.openstack.org/#/c/517105/ should have the appropriate tests now to exercise the code in question. thanks for the assist! | 19:53 |
| clarkb | SamYaple: thanks I'll take a look in a bit | 19:53 |
| SamYaple | no rush | 19:53 |
| dmsimard | mriedem, superdan: I created https://bugs.launchpad.net/nova/+bug/1735823 .. adding some more info as a comment | 19:54 |
| openstack | Launchpad bug 1735823 in OpenStack Compute (nova) "Nova can hang when creating a VM with file injection" [Undecided,New] | 19:54 |
| mriedem | https://github.com/eventlet/eventlet/blob/v0.20.0/eventlet/tpool.py#L155-L160 | 19:54 |
| superdan | hanging during file injection helps our case for killing it with fire, right? | 19:54 |
| mriedem | superdan: yeah | 19:55 |
| mriedem | but, | 19:55 |
| mriedem | my api change doesn't remove file injection, you can still do it via v2.1 | 19:55 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Make sure we collect var/lib/heat-config directory. https://review.openstack.org/523388 | 19:55 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Introduce TRIPLEO_HEAT_TEMPLATES_ROOT https://review.openstack.org/523488 | 19:55 |
| superdan | boo | 19:55 |
| mriedem | this isn't personality files either, it's just the inject_partition flag | 19:55 |
| mriedem | set to -1 | 19:55 |
| mriedem | we could put a big fat warning in the inject_partition option help saying, "anything but -2 (disable) might hang your compute" | 19:56 |
| mriedem | well, not your host, but could hang a request | 19:56 |
| *** dklyle has quit IRC | 19:57 | |
| superdan | or | 20:00 |
| superdan | "this fragile feature is ... fragile" | 20:00 |
| superdan | "The warranty for this feature expired in 2012" | 20:00 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Enforce a minimum TTL for DNS records in TripleO jobs https://review.openstack.org/524018 | 20:00 |
| fungi | infra-puppet-core: trying to launch a server instantiated from our ::mailman module, it's failing to bootstrap because apache complains "Invalid command 'RewriteEngine', perhaps misspelled or defined by a module not included in the server configuration" so should we be declaring a require relationship in the ::httpd::vhost on the httpd_mod resources at | 20:05 |
| fungi | http://git.openstack.org/cgit/openstack-infra/puppet-mailman/tree/manifests/init.pp ? | 20:05 |
| mordred | fungi: probably? | 20:05 |
| dirk | mriedem: bit you're saying the guestfs proxy just hangs because guestfs hangs? Or is there some fundamental problem with the proxy and guestfs itself works fine? | 20:05 |
| mordred | superdan, mriedem: I know I don't actually get a vote, but I vote for killing it with fire | 20:05 |
| mriedem | idk, but from the logs i can see where we "stop" effectively on the one request, and the logs context switch, | 20:05 |
| *** weshay_bbiab is now known as weshay | 20:06 | |
| mriedem | so i assume it's a problem with eventlet switching here | 20:06 |
| mriedem | i don't know enough about http://eventlet.net/doc/threading.html#tpool-simple-thread-pool to say if there is a better way to do what it's doing | 20:06 |
| dmsimard | In EL7 the version of guestfs is libguestfs-1.36.3-6.el7_4.3.x86_64 .. in Ubuntu... I'm not seeing it installed? | 20:07 |
| *** camunoz has quit IRC | 20:07 | |
| *** mnencia has quit IRC | 20:07 | |
| mordred | dmsimard: it's possible that feature isn't enabled in the ubuntu jobs? (which would also explain why we only see the problem on non-ubuntu) | 20:08 |
| *** sree has joined #openstack-infra | 20:08 | |
| dmsimard | mordred: devstack jobs on centos and ubuntu would run a different set ? /me looks | 20:08 |
| mriedem | no we do file injection on ubuntu jobs too | 20:08 |
| mriedem | http://logs.openstack.org/10/523910/1/check/legacy-tempest-dsvm-neutron-full/8cd6149/logs/etc/nova/nova.conf.txt.gz | 20:08 |
| mriedem | inject_partition = -1 | 20:08 |
| mriedem | http://logs.openstack.org/10/523910/1/check/legacy-tempest-dsvm-neutron-full/8cd6149/logs/dpkg-l.txt.gz | 20:09 |
| mriedem | ii libguestfs0:amd64 1:1.32.2-4ubuntu2 amd64 guest disk image management system - shared library | 20:09 |
| mordred | ah. k | 20:09 |
| mordred | darn. I was hoping it was more ammunition against the feature. /me goes back to other things | 20:09 |
| *** mnencia has joined #openstack-infra | 20:09 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mailman master: Enable modules before starting Apache https://review.openstack.org/524718 | 20:11 |
| fungi | mordred: ^ | 20:11 |
| dmsimard | mriedem: so there's a delta between 1.32.2 in Ubuntu and 1.36.3 in CentOS... and opensuse has 1.32.4 | 20:11 |
| *** weshay is now known as weshay_interview | 20:12 | |
| *** sree has quit IRC | 20:12 | |
| mordred | fungi: that seems completely reasonable | 20:13 |
| fungi | mordred: i copied that workaround from another one of our puppet modules, so it must be okay ;) | 20:13 |
| mordred | fungi: when has the cult of cargo ever led us astray? | 20:13 |
| dirk | dmsimard: if it is only happening on OVH this sounds like an issue with the guestfs kernel rather that guestfs itself | 20:13 |
| dmsimard | dirk: it's pretty low level that's for sure, and worth communicating up to the ovh ops once we have a better idea | 20:14 |
| *** ldnunes has quit IRC | 20:14 | |
| dmsimard | ovh is known to run custom kernels so maybe that is biting us right now, I don't know | 20:14 |
| clarkb | does libguestfs use virt to do its job? | 20:15 |
| fungi | mordred: vive le culte de la cargaison! | 20:15 |
| clarkb | because that could explain why it affects ovh | 20:15 |
| dirk | clarkb: it uses qemu-system-x86 directly afaik | 20:15 |
| dirk | Is there a way to add an ssh key to an instance running on ovh that has that issue? | 20:16 |
| clarkb | that may actually explain ti then | 20:16 |
| clarkb | we know nested virt doesn't work on ovh | 20:16 |
| clarkb | if libguestfs virts then boom | 20:16 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't set job var override_checkout if null https://review.openstack.org/524414 | 20:16 |
| dmsimard | clarkb: you mean libguestfs tries to spawn a vm with the image but does it with kvm instead of qemu ? | 20:17 |
| dmsimard | (in order to do the file injection) | 20:17 |
| dirk | clarkb: I am not sure if it does usermode emulation or enables nested KVM | 20:17 |
| clarkb | dmsimard: ya | 20:17 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix scheduler reconfiguration handler https://review.openstack.org/524714 | 20:17 |
| dirk | We could just run guestfs-test in devstack prior to enabling thr feature | 20:18 |
| dirk | I bet it will hang/crash/explode | 20:18 |
| *** Goneri has joined #openstack-infra | 20:18 | |
| *** trown|lunch is now known as trown | 20:19 | |
| clarkb | ovh was interested in debugging the nested virt issues further, but I've not had time to work with rm_work and compile a reproduction case for them | 20:19 |
| clarkb | also there is no guaruntee they would be able to fix it either | 20:19 |
| clarkb | since stuff like this tends to be veryhardware specific | 20:19 |
| *** thorst has quit IRC | 20:19 | |
| rm_work | hmm | 20:20 |
| rm_work | well the repro is pretty easy | 20:20 |
| rm_work | i can put up a patch | 20:20 |
| rm_work | if you'd like | 20:20 |
| *** dprince has quit IRC | 20:21 | |
| dmsimard | clarkb: nested virt or not, we agree that this is not occurring elsewhere right ? neither rax with Xen etc | 20:21 |
| fungi | hard to prove a negative, but so far i've seen no _evidence_ of it transpiring elsewhere | 20:22 |
| clarkb | export LIBGUESTFS_BACKEND_SETTINGS=force_tcg will force not using kvm | 20:23 |
| *** slaweq has joined #openstack-infra | 20:23 | |
| clarkb | dmsimard: correct because elsewhere does not have nested virt (rax) or has working nested virt (osic, not sure of any currently with working nested virt) | 20:23 |
| rm_work | it does not happen elsewhere | 20:24 |
| clarkb | based on the above env setting I think it will use kvm by default | 20:24 |
| rm_work | trust me, we'd see it | 20:24 |
| rm_work | octavia forces kvm on if it is supported | 20:24 |
| dmsimard | clarkb: I'm testing locally to see what behavior | 20:24 |
| rm_work | and only OVH hosts ever had issues | 20:24 |
| *** fried_rolls is now known as fried_rice | 20:24 | |
| dmsimard | rm_work: you're seeing this too ? | 20:24 |
| rm_work | if any other clouds had issues, we would see spurious failures on octavia jobs | 20:24 |
| rm_work | which issue specifically are you seeing? | 20:24 |
| dmsimard | rm_work: https://bugs.launchpad.net/nova/+bug/1735823 | 20:25 |
| openstack | Launchpad bug 1735823 in OpenStack Compute (nova) "Nova can hang when creating a VM with disk injection" [Undecided,New] | 20:25 |
| rm_work | i'm referring to OVH kvm crashing when trying to boot VMs | 20:25 |
| *** armaan has joined #openstack-infra | 20:25 | |
| clarkb | http://libguestfs.org/guestfs.3.html#backend-settings is where that is documented | 20:25 |
| *** pcaruana has quit IRC | 20:25 | |
| rm_work | that's possibly the cause | 20:25 |
| rm_work | johnsom: ^^ | 20:25 |
| rm_work | we don't know the real cause, it COULD be that | 20:25 |
| rm_work | does it also happen in Ubuntu? | 20:25 |
| dmsimard | clarkb: I'm downloading $randomcloudimage to test and will report back | 20:25 |
| rm_work | because we use Ubuntu on the gate | 20:25 |
| clarkb | rm_work: no | 20:25 |
| rm_work | hmm ok | 20:25 |
| rm_work | then it's a different thing | 20:25 |
| clarkb | rm_work: not necessarily | 20:25 |
| rm_work | well, could be related i guess | 20:26 |
| clarkb | ubuntu could patch their lib guestfs or any other number of things | 20:26 |
| rm_work | but we see our issue in ubuntu | 20:26 |
| dmsimard | rm_work: we haven't *seen* it happen in ubuntu, if you have an occurence please do share | 20:26 |
| clarkb | yes ovh virt issue is fairly universal | 20:26 |
| clarkb | dmsimard: their issue manifests differently | 20:26 |
| rm_work | well, how do you prove it's THAT | 20:26 |
| dmsimard | mriedem found where it hangs | 20:26 |
| clarkb | lets back up | 20:27 |
| clarkb | octavia has problems with nested virt in ovh | 20:27 |
| *** ykarel|away has quit IRC | 20:27 | |
| clarkb | other people have had problems with nested virt in ovh as well | 20:27 |
| clarkb | (like trove) | 20:27 |
| dirk | dmsimard: just run the guestfs Selftest | 20:27 |
| dirk | In a test DNM review and we'll see | 20:27 |
| clarkb | infra doing independent testing has also found problems with nested virt in ovh it is what caused us to disable nested virt in devstack-gate | 20:27 |
| dmsimard | rm_work: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-12-01.log.html#t2017-12-01T19:43:34 | 20:27 |
| superdan | are you guys seeing this hang in ovh proper, or devstack's nova's use of libguestfs? | 20:27 |
| *** slaweq has quit IRC | 20:27 | |
| clarkb | superdan: devstack nova use of libguestfs running on ovh VMs | 20:28 |
| superdan | right, okay | 20:28 |
| clarkb | we know with a high degree of certainty that nested virt will not work today in ovh regardless of distro or job or why nested virt is being used | 20:28 |
| rm_work | dmsimard: not sure that i've seen that... would have to try to dig up old logs | 20:28 |
| superdan | and you're thinking that guestfs is using nested virt when it starts up its service vm, | 20:28 |
| superdan | and that that is broken on ovh more than others? | 20:28 |
| clarkb | superdan: yes that is my theory | 20:28 |
| clarkb | superdan: and reading guestfs docs seems to support that as there is a toggle to turn off virt | 20:28 |
| superdan | is ovh the only one with proper nested virt enabled? | 20:29 |
| clarkb | superdan: osic had it, not sure of current clouds | 20:29 |
| mriedem | there is something about force_tcg in nova... | 20:29 |
| mriedem | danpb added | 20:29 |
| superdan | right, so my understanding is that nested virt with kvm is still not stable in any way | 20:29 |
| clarkb | superdan: on Intel, correct | 20:29 |
| clarkb | with AMD cpus its apparently enabled by default in a default linux kernel build | 20:29 |
| clarkb | with Intel it is disabled by default in ^ | 20:29 |
| superdan | not sure that means it's stable and haven't heard exclusions for AMD for our support, but .. okay | 20:30 |
| clarkb | its stable enough the linux kernel thinks it won't break users | 20:30 |
| * superdan points to his previous statement | 20:30 | |
| superdan | AFAIK, _we_ won't support it anywhere | 20:30 |
| mriedem | superdan: clarkb: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L474-L478 | 20:30 |
| *** wolverineav has quit IRC | 20:30 | |
| *** wolverineav has joined #openstack-infra | 20:31 | |
| fungi | probably means it's hard to write our documentation with good recommendations to turn on nested virt only on certain manufacturers' machines (and especially given that a server vendor might use different cpu vendors in similar models) | 20:31 |
| superdan | is ovh running AMD cpus? | 20:31 |
| clarkb | mriedem: neat, so fi that is working it would contradict my theory | 20:31 |
| dmsimard | in those regions, no | 20:31 |
| clarkb | superdan: no Intel I think | 20:32 |
| superdan | okay | 20:32 |
| dmsimard | intel e5's | 20:32 |
| fungi | clarkb: related theory... could a regression in some libguestfs versions have caused it to start ignoring that option? ;) | 20:33 |
| clarkb | fungi: or we are setting the wrong option in nova beacuse they changed it, ya that could be | 20:33 |
| dmsimard | dirk: output from guestfs-test http://paste.openstack.org/raw/627993/ | 20:34 |
| fungi | my motto: never trust computers | 20:34 |
| dmsimard | dirk: on a random VM at home, not on OVH or anything | 20:34 |
| clarkb | https://github.com/openstack/nova/blob/45dfc7106ebb95bacc2464ff37f372aae785691d/nova/virt/disk/vfs/guestfs.py#L196-L201 we explicitly fail to set it in some cases appaerntly | 20:34 |
| *** armaan_ has joined #openstack-infra | 20:34 | |
| fungi | oh, that's swell | 20:35 |
| clarkb | that log warning doesn't show up in the nova compute log from this job though | 20:36 |
| *** rossella_s has quit IRC | 20:36 | |
| superdan | we could blacklist the kvm module in devstack before we start doing things right? | 20:36 |
| *** armaan has quit IRC | 20:36 | |
| *** armaan_ has quit IRC | 20:37 | |
| mriedem | clarkb: i think that would be really old versions at this point | 20:37 |
| mriedem | that patch is 4 years old now | 20:37 |
| mriedem | that warning doesn't show up in logstash either | 20:37 |
| *** rossella_s has joined #openstack-infra | 20:38 | |
| dirk | dmsimard: sure. I meant the ovh worker ;-) | 20:39 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-openstackci master: Lay groundwork for zuulv2/v3 coexisting https://review.openstack.org/523951 | 20:42 |
| clarkb | mriedem: rtfsing python guestfs shows that set_backend_settings takes a list and resets all settings. set_backend_setting sets a single setting and does not affect other settings | 20:42 |
| clarkb | mriedem: it is possible the use of set_backend_settings is overwriting things if called multiple times or the string arg isn't quite right for the method? | 20:43 |
| superdan | clarkb: why not just blacklist kvm in our node setup? | 20:43 |
| clarkb | superdan: the kernel module you mean? | 20:43 |
| superdan | yeah | 20:43 |
| *** dprince has joined #openstack-infra | 20:43 | |
| clarkb | we can' but every tool tries really hard to undo it for you | 20:43 |
| clarkb | devstack for example will go so far as to install packaging and insmod it | 20:44 |
| superdan | if we blacklist it in our own modprobe.conf.d I would think we'd be okay | 20:44 |
| clarkb | so it becomes a game of whack-a-mole to track it all down | 20:44 |
| clarkb | will insmod honor modprobe settings? | 20:44 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Ensure ChangeLog exists in install-if-python https://review.openstack.org/524712 | 20:44 |
| superdan | I think so | 20:44 |
| clarkb | I thought insmod was the I don't care about your settings method | 20:44 |
| superdan | we could also: | 20:44 |
| superdan | 1. nuke/move the .ko | 20:44 |
| dmsimard | Devstack works with kvm, I wouldn't hardcode that in | 20:45 |
| superdan | 2. Create and mangle /dev/kvm | 20:45 |
| superdan | dmsimard: no it'd be a flag or done ahead of time before devstack | 20:45 |
| dmsimard | Right, I wonder if it would create problems (or hide others) | 20:45 |
| superdan | well, it's the way we make sure our environment isn't nested regardless of where we're running | 20:46 |
| clarkb | mriedem: superdan in any case I'm not expert but https://github.com/openstack/nova/blob/45dfc7106ebb95bacc2464ff37f372aae785691d/nova/virt/disk/vfs/guestfs.py#L195 looks potentially buggy to me | 20:46 |
| clarkb | let my copy pasta some guestfs code for comparison | 20:46 |
| superdan | if we know nested virt is very broken, we really should be making sure our environment isn't nested | 20:46 |
| clarkb | mriedem: superdan http://paste.openstack.org/show/627994/ | 20:46 |
| clarkb | superdan: I agree and so far what we have done has worked | 20:47 |
| superdan | ensuring we disable it everywhere in the code is also whack-a-mole | 20:47 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-openstackci master: Lay groundwork for zuulv2/v3 coexisting https://review.openstack.org/523951 | 20:47 |
| clarkb | it would be funny if the the set_backend_settings() call in old libguestfs (eg ubuntu) did what nova wants there but in newer one does not | 20:47 |
| clarkb | the version I pasted aboev is from the code that should be in centos 7 based on the version | 20:48 |
| fungi | in an occam's razor sense of the term "funny" anyway | 20:48 |
| mriedem | clarkb: yeah that looks totally wrong from what we're doing | 20:48 |
| johnsom | clarkb I can give details on the OVH issue. Currently we have disabled KVM to work around. The upstream kernel bug is here: https://bugzilla.kernel.org/show_bug.cgi?id=192521 | 20:48 |
| openstack | bugzilla.kernel.org bug 192521 in kvm "KVM: entry failed, hardware error 0x0" [High,New] - Assigned to virtualization_kvm | 20:48 |
| mriedem | we should be able to do a hasattr on the handle to see if 'set_backend_setting' is there | 20:48 |
| johnsom | Of course rm_work would ping me while I ran out for a burrito... | 20:49 |
| mriedem | clarkb: looking at this though | 20:49 |
| mriedem | http://libguestfs.org/guestfs.3.html#backend-settings | 20:49 |
| mriedem | it looks like set_backend_settings just takes a list, e.g. export LIBGUESTFS_BACKEND_SETTINGS=force_tcg | 20:49 |
| mriedem | but list("force_tcg") would be wrong | 20:50 |
| *** thorst has joined #openstack-infra | 20:50 | |
| *** ralonsoh has quit IRC | 20:50 | |
| mriedem | unless ['f', 'o', 'r', 'c', 'e', '_', 't', 'c', 'g'] works :) | 20:50 |
| clarkb | mriedem: ya thats what I'm wondering | 20:50 |
| johnsom | The only other nova/qemu bug I am aware of is related to devstack pulling in the ocata or pike apt repo which pulls in a version of qemu that has defaults that fail on xenial. You have to set a machine type in nova to get instances to boot. "[libvirt] hw_machine_type x86_64=pc-i440fx-xenial" | 20:51 |
| mriedem | This function returns 0 on success or -1 on error. | 20:51 |
| mriedem | clarkb: we can check for a return code and log an error to find out | 20:51 |
| *** rossella_s has quit IRC | 20:51 | |
| *** jkilpatr has joined #openstack-infra | 20:51 | |
| johnsom | Otherwise all of the other hosts that had nested virt enabled had no issues (osic, bluebox, etc.). In fact, OVH didn't have the issue until something changed there. | 20:52 |
| openstackgerrit | Merged openstack-infra/puppet-mailman master: Enable modules before starting Apache https://review.openstack.org/524718 | 20:52 |
| clarkb | johnsom: thats interesting becuase we totally had to change to the UCA libvirt to get VMs taht worked :) | 20:52 |
| clarkb | johnsom: there were significant numbers of memory related failures causing VM craashes with stock xenial | 20:53 |
| johnsom | clarkb Yeah, understand, just different situations will cause qemu to 100% cpu spin and not boot. | 20:53 |
| clarkb | johnsom: I wonder if your images require more cpu instructions than others | 20:53 |
| johnsom | clarkb Doubt it, it's stock ubuntu cloud image as pulled in via diskimage-builder | 20:54 |
| *** rossella_s has joined #openstack-infra | 20:54 | |
| superdan | clarkb: devstack doesn't have any mention of insmod that I can see, and anything really should be using modprobe, else it needs the path to the module from the current kernel, which would be kinda weird | 20:55 |
| superdan | clarkb: but insmod does indeed ignore config (despite having symbols that makes it look like it might) | 20:55 |
| *** thorst has quit IRC | 20:55 | |
| superdan | anyway, ya'lls call, but I'd be chasing that first personally | 20:56 |
| clarkb | superdan: ya devstack does indeed do a modprobe not an insmod (never trust my memory I guess) | 20:56 |
| clarkb | superdan: so your suggestion would be to modify modprobe config to blacklist kvm, then remove the module? | 20:56 |
| superdan | clarkb: echo "install kvm /bin/false" > /etc/modprobe.d/openstack-infra.conf; rmmod kvm_intel; rmmod kvm | 20:57 |
| *** sree has joined #openstack-infra | 20:57 | |
| superdan | if nothing has /dev/kvm open when you're doing that you should be good | 20:57 |
| superdan | not having it loaded makes all our environments a lot more similar for any other feature probing that might be done if kvm is enabled, which seems like a good idea, although I know that's a little bit of a stretch | 20:57 |
| mriedem | clarkb: https://review.openstack.org/524727 | 20:58 |
| clarkb | superdan: ya, also johnsom/rm_work would revolt if we made it too difficult for them to be more opinionated | 20:58 |
| clarkb | superdan: but I think in the general case we can likely get away with something like that | 20:58 |
| clarkb | however things like packstack likeyl won't be affected by devstack gate | 20:58 |
| clarkb | dmsimard: ^ you're jobs aren't using devstack-gate are they? | 20:59 |
| johnsom | Yeah, enabling KVM takes ~50 minutes off a tempest runtime | 20:59 |
| superdan | clarkb: johnsom rm_work: I'd do this in node setup where we know what kind of node we have and that we want it, | 20:59 |
| superdan | not in devstack or anything like that | 20:59 |
| *** jascott1 has quit IRC | 20:59 | |
| clarkb | ya we could add a thing that does it to all ovh nodes in the base job or something | 21:00 |
| superdan | I know I said devstack earlier, but I meant "in setup somewhere" | 21:00 |
| superdan | yeah | 21:00 |
| clarkb | that would be early enough that nothing should be using the kvm module either | 21:00 |
| johnsom | That is what we do. Our gate checks if the host has the proper cpu flags and enables based on that. However we had to add a OVH check and disable because it has the proper flags, but KVM crashes. | 21:00 |
| *** jascott1 has joined #openstack-infra | 21:01 | |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/tests/contrib/gate_hook.sh#L32 | 21:01 |
| *** sree has quit IRC | 21:02 | |
| johnsom | obviously we haven't made these completely zuulv3 native yet... | 21:02 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Retire Packaging Deb project repos https://review.openstack.org/524730 | 21:03 |
| clarkb | dmsimard: trying out superdan's suggestion would be one way to check fi that fixes it. Basically make change, depends on it from packstack or wherever then recheck until job runs on ovh | 21:03 |
| superdan | if you disable kvm entirely (and it stays off) and that doesn't fix the problem, then.. bigger things are afoot for sure | 21:04 |
| johnsom | The above gate hook code works. It's just a shame we have to exclude OVH. Switch off KVM and back to TCG makes nova boot (full kernel booted time) go from about 30 seconds to 8-10 minutes. | 21:08 |
| openstackgerrit | Eric Fried proposed openstack-infra/os-loganalyze master: Strip ANSI color codes https://review.openstack.org/524731 | 21:09 |
| fried_rice | jeblair fungi This one might be more likely to fly ^ | 21:09 |
| fried_rice | It's a no-op if there are no color codes to begin with. | 21:09 |
| *** ianw has joined #openstack-infra | 21:09 | |
| *** slaweq has joined #openstack-infra | 21:10 | |
| johnsom | I'm going to disappear back into my openstack-lbaas world, but if I can help or answer questions ping me. Willing to help repro or work with folks. | 21:10 |
| johnsom | clarkb One more detail I forgot, the KVM crash occurs with the stock cirros image as well as the ubuntu cloud image. | 21:17 |
| clarkb | johnsom: ya trove's images too | 21:17 |
| johnsom | Yep | 21:18 |
| *** linkmark has joined #openstack-infra | 21:18 | |
| *** weshay_interview is now known as weshay | 21:19 | |
| *** ijw has joined #openstack-infra | 21:19 | |
| *** thorst has joined #openstack-infra | 21:20 | |
| *** ntpttr_ has joined #openstack-infra | 21:21 | |
| *** rcernin has joined #openstack-infra | 21:22 | |
| dmsimard | johnsom: so wait, you're saying you're actually using KVM all the time ? Unless it's OVH ? | 21:23 |
| clarkb | or rax | 21:23 |
| johnsom | dmsimard Yes, for about two years now | 21:23 |
| clarkb | or any other cloud that doesn't support virt | 21:23 |
| clarkb | I haven't checked internap or city recently but I thought internap may have it disabled | 21:23 |
| johnsom | The OVH exception was just this year, prior it ran fine | 21:24 |
| dmsimard | clarkb: I wonder if it would be worth considering creating a label and using that when we're interested in nested | 21:24 |
| dmsimard | (in nodepool) | 21:24 |
| *** smatzek_ has quit IRC | 21:25 | |
| clarkb | dmsimard: we've explicityl said no to that beause we can't reliably offer it | 21:25 |
| clarkb | many of the clouds we have used over time do not support it | 21:25 |
| clarkb | and nothing says the cluods we use that do support it today will continue to support it | 21:25 |
| dmsimard | I mean, I've always found it weird that we don't reliably test KVM upstream.. no one goes to production with qemu in their right mind | 21:25 |
| *** thorst has quit IRC | 21:25 | |
| johnsom | dmsimard +1 to that | 21:25 |
| clarkb | yes its a concern but really a minor one | 21:26 |
| clarkb | we aren't testing kvm | 21:26 |
| clarkb | we are testing nova | 21:26 |
| clarkb | and 99% of nova is the same regardless | 21:26 |
| clarkb | (I don't actaully know what the number is but the delta between qemu and kvm should be very small) | 21:26 |
| dmsimard | Bah, you mean it's abstracted by libvirt ? | 21:26 |
| clarkb | yes | 21:26 |
| clarkb | if our job was to test kvm I would care far more | 21:27 |
| superdan | +1 | 21:27 |
| * johnsom needs a beer to have the philosophical discussion of end-to-end testing again... grin | 21:27 | |
| dmsimard | I understand that, but I'm just saying no one goes to production with qemu, but then again, we could say the same for devstack :p | 21:28 |
| clarkb | dmsimard: also you keep saying that like the tripleo cloud wasn't running a bunch of qemu hypervisors recently >_> | 21:28 |
| superdan | and nobody is running nested virt for their primary instances | 21:28 |
| superdan | so it's all fake anyway | 21:28 |
| dmsimard | fair | 21:28 |
| superdan | if we were running on metal, I'd totally say we should be using kvm guests | 21:29 |
| superdan | but instead we're running something we _know_ is not how things work in production and is known broken | 21:29 |
| clarkb | there are potentially valid reasons to qemu in production too, like testing arm without arm hardware | 21:29 |
| dmsimard | clarkb: I don't follow, what cloud ? the ones I know of run kvm from bare metal | 21:29 |
| superdan | sure | 21:29 |
| openstackgerrit | Eric Fried proposed openstack-infra/os-loganalyze master: Strip ANSI color codes https://review.openstack.org/524731 | 21:29 |
| clarkb | dmsimard: a few weeks back there was a thing about how tripleo jobs were slow in part due to some hypervisors in the red hat tripleo cloud being configured to qemu not kvm | 21:29 |
| dmsimard | clarkb: vaguely rings a bell :/ | 21:30 |
| * dmsimard is on too many threads | 21:30 | |
| dmsimard | but what I was going on about reminds me that I meant to ask.. do we have documentation on what we consider a release "done" ? Is there a list of criteria must be met ? | 21:31 |
| *** thorst has joined #openstack-infra | 21:31 | |
| clarkb | dmsimard: like an openstack release? | 21:32 |
| dmsimard | yeah | 21:32 |
| dmsimard | what makes the go/no-go, devstack must pass, etc | 21:32 |
| clarkb | I think the release team has that written down somewhere. AIUI they are date based with a period of RCs and as long as RCs have no major world breaking bugs that are known about the RCs are promoted to releases on release day | 21:32 |
| dmsimard | ok so it's up to the discretion of each individual project to some extent | 21:33 |
| dmsimard | as to what they consider world breaking bugs | 21:33 |
| clarkb | https://releases.openstack.org/reference/release_models.html ya I don't think its encoded as strictly as these jobs must pass (though any gating job must pass because thats how you make a release) | 21:34 |
| clarkb | by definition of gating | 21:34 |
| *** ociuhandu has joined #openstack-infra | 21:34 | |
| *** thorst has quit IRC | 21:36 | |
| dmsimard | mriedem: thanks for your help with this today, much appreciated :D | 21:38 |
| mriedem | np | 21:38 |
| *** rlandy has quit IRC | 21:39 | |
| dmsimard | clarkb: omg the irony: mriedem's patch to enable guestfs debug reproduced the centos issue on devstack: https://review.openstack.org/#/c/524710/ && http://logs.openstack.org/10/524710/1/check/legacy-tempest-dsvm-neutron-full-centos-7/e8eae03/job-output.txt.gz#_2017-12-01_21_13_15_415131 | 21:39 |
| * dmsimard goes searching for logs | 21:40 | |
| *** rossella_s has quit IRC | 21:40 | |
| *** ociuhandu has quit IRC | 21:40 | |
| dmsimard | I'm going to open a bug with the new FF57, some of those heavy log pages hangs my whole window :/ | 21:41 |
| *** rossella_s has joined #openstack-infra | 21:42 | |
| *** jcoufal has quit IRC | 21:43 | |
| *** ianychoi has quit IRC | 21:44 | |
| dmsimard | nova-compute: http://logs.openstack.org/10/524710/1/check/legacy-tempest-dsvm-neutron-full-centos-7/e8eae03/logs/screen-n-cpu.txt.gz#_Dec_01_20_24_19_820703 | 21:47 |
| dmsimard | libvirt: http://logs.openstack.org/10/524710/1/check/legacy-tempest-dsvm-neutron-full-centos-7/e8eae03/logs/screen-n-cpu.txt.gz#_Dec_01_20_24_19_820703 | 21:47 |
| fungi | at least it didn't take long to reproduce? ;) | 21:48 |
| *** thorst has joined #openstack-infra | 21:49 | |
| * fungi grumbles at the discovery that we have checks against our gerrit/projects.yaml to insist on a repo description even if it's a retired repo | 21:49 | |
| jeblair | fungi: seems like an especially good place to have a description... like "retired in favor of ..." would be ideal :) | 21:50 |
| openstackgerrit | Eric Fried proposed openstack-infra/os-loganalyze master: Strip ANSI color codes https://review.openstack.org/524731 | 21:50 |
| fungi | yeah, more that i have to redo the patch which adjusts the acls for hundreds of retired packaging-deb repos | 21:51 |
| jeblair | fungi: i heard you were a robot | 21:51 |
| dmsimard | the libvirt debug logs are... wow | 21:51 |
| fungi | i wonder if i should replace their descriptions rather than restore them | 21:51 |
| jeblair | fungi: ++ | 21:51 |
| fungi | i think i will boilerplate them | 21:52 |
| fungi | yeah | 21:52 |
| mriedem | dmsimard: you can see the guestfs thread hung when things dump http://logs.openstack.org/10/524710/1/check/legacy-tempest-dsvm-neutron-full-centos-7/e8eae03/logs/screen-n-cpu.txt.gz#_Dec_01_21_13_22_230962 | 21:52 |
| mriedem | at the end | 21:52 |
| dmsimard | wow | 21:53 |
| mriedem | right here https://github.com/openstack/nova/blob/45dfc7106ebb95bacc2464ff37f372aae785691d/nova/virt/disk/vfs/guestfs.py#L80 | 21:54 |
| mriedem | so that's definitely the hang | 21:54 |
| mriedem | i mean, 95% definitly | 21:54 |
| dmsimard | well all the traces stop there | 21:56 |
| dmsimard | g.add_drive("/dev/null") ? | 21:56 |
| mriedem | http://libguestfs.org/guestfs.3.html#guestfs_add_drive | 21:57 |
| mriedem | i don't really know why that's there | 21:58 |
| mriedem | i think it has something to do with that call just being a test to see that guestfs works | 21:58 |
| mriedem | something noop | 21:58 |
| *** smatzek has joined #openstack-infra | 21:59 | |
| dmsimard | mriedem: yeah but the documentation also says that you shouldn't do that if you need to read or write to it | 21:59 |
| dmsimard | http://libguestfs.org/guestfs.3.html -> NULL DISKS | 21:59 |
| dmsimard | but higher up in the docs there's an example that does exactly that so I guess it's expected to work | 21:59 |
| *** smatzek has quit IRC | 22:00 | |
| *** smatzek has joined #openstack-infra | 22:00 | |
| mriedem | http://libguestfs.org/guestfs-performance.1.html | 22:00 |
| mriedem | yeah, says it's used for testing | 22:01 |
| *** smatzek has quit IRC | 22:01 | |
| mriedem | If you want to use libguestfs APIs that don’t refer to disks, since libguestfs requires that at least one disk is added, you should add a null disk. | 22:01 |
| *** trown is now known as trown|outtypewww | 22:01 | |
| *** smatzek has joined #openstack-infra | 22:01 | |
| mriedem | i'm not sure why we don't have a finally block that closes g.close() | 22:02 |
| mriedem | maybe not needed | 22:02 |
| *** thorst has quit IRC | 22:03 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: WIP Create a tets matrix role https://review.openstack.org/524402 | 22:04 |
| *** iyamahat_ has joined #openstack-infra | 22:05 | |
| *** iyamahat has quit IRC | 22:05 | |
| mriedem | i'm not seeing https://review.openstack.org/#/c/524727/1/nova/virt/disk/vfs/guestfs.py in test runs so far, so it seems we are forcing tcg mode | 22:05 |
| dmsimard | you looked with logstash ? | 22:06 |
| *** smatzek has quit IRC | 22:06 | |
| *** smatzek has joined #openstack-infra | 22:07 | |
| dmsimard | nevermind -- clarkb: the logstash looks clean now! | 22:07 |
| dmsimard | logstash queue* | 22:07 |
| *** flwang has joined #openstack-infra | 22:07 | |
| dmsimard | http://grafana.openstack.org/dashboard/db/zuul-status?panelId=19&fullscreen&from=1509508800000&to=1512104399999 | 22:08 |
| mriedem | well, i don't know if there might be a difference in the centos jobs, | 22:09 |
| mriedem | we don't run those on nova | 22:09 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/project-config master: Make stable jobs voting on Tempest https://review.openstack.org/524741 | 22:09 |
| *** smatzek has quit IRC | 22:10 | |
| *** smatzek has joined #openstack-infra | 22:10 | |
| andreaf | mtreinish, dmsimard, fungi: ^^^ this was lost in the zuulv3 migration we should restore the voting status of stable jobs on tempest | 22:10 |
| *** Goneri has quit IRC | 22:10 | |
| dmsimard | andreaf: added a comment | 22:11 |
| andreaf | jeblair: still wip, but I have now an example usage of the test-matrix role https://review.openstack.org/#/q/topic:test_matrix_role+(status:open+OR+status:merged) | 22:12 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/project-config master: Make stable jobs voting on Tempest https://review.openstack.org/524741 | 22:13 |
| andreaf | dmsimard: thanks - fixed ^^^ | 22:13 |
| *** smatzek has quit IRC | 22:14 | |
| clarkb | mriedem: or its writing out each character of the string each as its own setting? | 22:14 |
| mriedem | clarkb: no idea | 22:14 |
| *** smatzek has joined #openstack-infra | 22:14 | |
| mriedem | i could put a patch on top that sends it a list | 22:15 |
| mriedem | i also see guestfs debug logging now in the n-cpu logs in that devstack patch http://logs.openstack.org/10/524710/1/check/legacy-tempest-dsvm-neutron-full/a1867f6/logs/screen-n-cpu.txt.gz#_Dec_01_20_12_24_558620 | 22:15 |
| mriedem | not sure what to make of it | 22:15 |
| mriedem | http://logs.openstack.org/10/524710/1/check/legacy-tempest-dsvm-neutron-full/a1867f6/logs/screen-n-cpu.txt.gz#_Dec_01_20_11_15_766154 | 22:16 |
| mriedem | event=trace eh=0 buf='get_backend_setting "force_tcg"' array=[] | 22:16 |
| mriedem | hmm | 22:16 |
| mriedem | HA | 22:16 |
| mriedem | yo'ure right http://logs.openstack.org/10/524710/1/check/legacy-tempest-dsvm-neutron-full/a1867f6/logs/screen-n-cpu.txt.gz#_Dec_01_20_11_15_758749 | 22:16 |
| mriedem | event=trace eh=0 buf='set_backend_settings "f o r c e _ t c g"' array=[] | 22:17 |
| dmsimard | nice | 22:17 |
| clarkb | ya because python | 22:17 |
| dmsimard | so true | 22:17 |
| mriedem | ok patch coming up | 22:17 |
| dmsimard | no other language that does that comes to mind | 22:17 |
| *** smatzek has quit IRC | 22:19 | |
| fungi | #status log Launched a new Mailman server corresponding to https://review.openstack.org/524322 and filed to exclude its ipv4 address from spamhaus record PBL1665489 | 22:19 |
| openstackstatus | fungi: finished logging | 22:19 |
| *** smatzek has joined #openstack-infra | 22:22 | |
| dmsimard | jeblair: wow, it's actually 3 patches to rename the base-integration jobs :/ | 22:22 |
| *** dprince has quit IRC | 22:23 | |
| *** wolverineav has quit IRC | 22:23 | |
| *** smatzek has quit IRC | 22:24 | |
| *** wolverineav has joined #openstack-infra | 22:24 | |
| *** smatzek has joined #openstack-infra | 22:24 | |
| dmsimard | 1) Create jobs with new names in openstack-zuul-jobs, 2) Rename jobs in project-config's zuul-jobs project definition 3) Remove old jobs from openstack-zuul-jobs and rename inside openstack-zuul-jobs project definition | 22:25 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Retire Packaging Deb project repos https://review.openstack.org/524730 | 22:26 |
| *** ijw has quit IRC | 22:27 | |
| jeblair | dmsimard: yeah, renaming is easy. moving is hard. that takes 5. | 22:29 |
| *** smatzek has quit IRC | 22:29 | |
| *** wolverineav has quit IRC | 22:29 | |
| dmsimard | :( | 22:29 |
| openstackgerrit | Eric Fried proposed openstack-infra/os-loganalyze master: Color themes for HTML view https://review.openstack.org/524744 | 22:30 |
| fried_rice | jeblair fungi ^ | 22:34 |
| mriedem | dmsimard: https://www.redhat.com/archives/libguestfs/2016-August/msg00218.html | 22:35 |
| fried_rice | See it in action here: http://184.172.12.213/manual/htmlify_logs/logs/n-cpu.txt.gz (new theme options below the sevs) | 22:35 |
| *** smatzek has joined #openstack-infra | 22:36 | |
| *** smatzek has quit IRC | 22:36 | |
| *** thorst has joined #openstack-infra | 22:40 | |
| *** sree has joined #openstack-infra | 22:40 | |
| *** ihrachys has quit IRC | 22:41 | |
| dmsimard | mriedem: right but is the fix as silly as replacing self.handle.set_backend_settings("force_tcg") by self.handle.set_backend_settings(["force_tcg"]) ? | 22:43 |
| dmsimard | The thread mentions direct (instead of force_tcg?) but you'd still have to pass it as a list ["direct"] because set_backend_settings expects a list | 22:43 |
| mriedem | dmsimard: we'll see https://review.openstack.org/524748 | 22:44 |
| mriedem | clarkb: ^ | 22:44 |
| dmsimard | The only place in all of OpenStack that it is used: http://codesearch.openstack.org/?q=set_backend_settings&i=nope&files=&repos= :D | 22:44 |
| *** sree has quit IRC | 22:45 | |
| *** iyamahat_ has quit IRC | 22:45 | |
| *** iyamahat has joined #openstack-infra | 22:45 | |
| *** thorst has quit IRC | 22:45 | |
| mriedem | testing here https://review.openstack.org/524750 | 22:45 |
| dmsimard | Now we need to roll the dice and get an OVH instance :D | 22:45 |
| clarkb | mriedem: you can also drop the s insettings | 22:45 |
| clarkb | that sets a single value | 22:45 |
| mriedem | clarkb: signature changes | 22:46 |
| mriedem | see the todo in my nova patch | 22:46 |
| johnsom | Hi folks. Can someone take a look at https://review.openstack.org/#/c/522666 and why it won't start in zuul? | 22:46 |
| dmsimard | Usually when that happens it's because there is a loop somewhere | 22:47 |
| *** mat128 has joined #openstack-infra | 22:47 | |
| johnsom | Ah, hmm, yeah I see he has a bunch of depends on in that patch. Let me look at that | 22:47 |
| dmsimard | Like a child patch doing a depends-on a parent patch so it loops | 22:47 |
| fungi | "dependency cycle" is the term we use in the zuul source | 22:48 |
| *** felipemonteiro_ has quit IRC | 22:48 | |
| johnsom | It doesn't log that anywhere for the user to see? | 22:48 |
| dmsimard | johnsom: I would suppose it's tricky because we don't want the loop to get started | 22:48 |
| dmsimard | I've asked about this before a long time ago, I forget why but there's a good reason | 22:49 |
| *** bobh has quit IRC | 22:49 | |
| johnsom | Yeah, he has some funky stuff in here. I can fix this. | 22:49 |
| fungi | in the past zuul avoided commenting on changes at all if it encountered a dependency cycle, as a safety measure. we've discussed possible ways to provide useful feedback but nobody's written anything yet | 22:49 |
| mriedem | dmsimard: ah we're already using a direct backend | 22:49 |
| mriedem | event=trace eh=0 buf='get_backend = "direct"' | 22:49 |
| dmsimard | well there you go, so it's probably just that then. | 22:50 |
| *** boden has quit IRC | 22:50 | |
| dmsimard | FWIW this made me read the libguestfs docs on python and wow, looks great :D | 22:51 |
| mriedem | failed to initialize KVM: No such file or directory | 22:51 |
| mriedem | Back to tcg accelerator | 22:51 |
| *** slaweq_ has joined #openstack-infra | 22:51 | |
| mriedem | so will be interesting to see if force_tcg makes guestfs stop trying to do kvm thingies | 22:51 |
| *** slaweq_ has quit IRC | 22:53 | |
| mriedem | warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] | 22:53 |
| dmsimard | mriedem: so wait, this means that previous version of libguestfs did NOT cast the string to a list ? | 22:53 |
| *** slaweq_ has joined #openstack-infra | 22:53 | |
| mriedem | dmsimard: i'm not sure about that | 22:53 |
| dmsimard | mriedem: because we're not encountering this bug on Ubuntu which happens to have an older version | 22:53 |
| mriedem | also, "warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]" makes me think that guestfs is figuring out that kvm isn't available, and it's doing tcg, | 22:54 |
| *** slaweq_ has quit IRC | 22:54 | |
| mriedem | and we still hit the hang because of an eventlet switch during launch | 22:54 |
| *** slaweq has quit IRC | 22:54 | |
| mriedem | but not sure | 22:54 |
| *** slaweq_ has joined #openstack-infra | 22:54 | |
| mriedem | this is really danpb territory | 22:54 |
| *** slaweq_ is now known as slaweq | 22:55 | |
| *** jtomasek has quit IRC | 22:55 | |
| dmsimard | johnsom: you said you were hitting the kvm issue with ovh on ubuntu right ? | 22:56 |
| johnsom | dmsimard Yes, the kvm crash was on a ubuntu instance booting both ubuntu and cirros nested instances. | 22:57 |
| johnsom | But only at OVH | 22:57 |
| *** mat128 has quit IRC | 22:57 | |
| *** jascott1 has quit IRC | 22:57 | |
| *** jascott1 has joined #openstack-infra | 22:58 | |
| dmsimard | found the bug in the backlog, nevermind it's not the same issue :( | 22:58 |
| *** jascott1 has quit IRC | 22:59 | |
| *** fried_rice is now known as efried | 22:59 | |
| *** jascott1 has joined #openstack-infra | 23:00 | |
| johnsom | dmsimard The CPU spin issue with qemu (not kvm crash) is related to the machine type being specified not matching the host. For xenial we set: [libvirt] hw_machine_type x86_64=pc-i440fx-xenial in nova.conf | 23:00 |
| *** jascott1 has quit IRC | 23:02 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: tox: remove validate-layout https://review.openstack.org/524753 | 23:02 |
| *** jascott1 has joined #openstack-infra | 23:02 | |
| *** slaweq has quit IRC | 23:03 | |
| *** slaweq has joined #openstack-infra | 23:04 | |
| *** slaweq has quit IRC | 23:04 | |
| *** slaweq has joined #openstack-infra | 23:05 | |
| *** jascott1 has quit IRC | 23:07 | |
| *** dhinesh_ has joined #openstack-infra | 23:08 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: tox: remove validate-layout https://review.openstack.org/524757 | 23:09 |
| *** dhinesh__ has joined #openstack-infra | 23:09 | |
| *** dhinesh_ has quit IRC | 23:09 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Move scenario001 and scenario003 back to voting https://review.openstack.org/524759 | 23:10 |
| dmsimard | current status: cloning north of 300MB from libguestfs's git repo :/ | 23:11 |
| *** dhinesh has quit IRC | 23:11 | |
| *** thorst has joined #openstack-infra | 23:14 | |
| *** ccamacho has quit IRC | 23:14 | |
| dmsimard | mriedem: doesn't look like the definition of the binding changed between 1.32 and 1.36 http://paste.openstack.org/raw/627999/ | 23:15 |
| dmsimard | so I don't know how that worked on Ubuntu | 23:15 |
| dmsimard | or why it wouldn't trigger the condition on OVH | 23:16 |
| *** dayou has quit IRC | 23:16 | |
| *** jascott1 has joined #openstack-infra | 23:19 | |
| *** thorst has quit IRC | 23:19 | |
| *** slaweq has quit IRC | 23:21 | |
| clarkb | distor patches? | 23:21 |
| dmsimard | hrm, let's see | 23:21 |
| dmsimard | they have 5 patches and it's mostly Makefile stuff. | 23:27 |
| *** mriedem has quit IRC | 23:29 | |
| *** salv-orlando has quit IRC | 23:37 | |
| *** wolverineav has joined #openstack-infra | 23:37 | |
| *** salv-orlando has joined #openstack-infra | 23:38 | |
| *** ganso has quit IRC | 23:41 | |
| *** salv-orlando has quit IRC | 23:42 | |
| jeblair | just sent this to #openstack-release: | 23:43 |
| jeblair | i think i recently introduced a bug in zuul that may break release pipelines. i'm currently adding more tests and working through the implications. hopefully we'll have it fixed by monday. | 23:43 |
| jeblair | infra-root, config-core: fyi ^ | 23:43 |
| mordred | jeblair: nod | 23:43 |
| *** greghaynes has quit IRC | 23:44 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Create ovb-ha-ipv6 and deprecate fs024 job https://review.openstack.org/523903 | 23:44 |
| *** thorst has joined #openstack-infra | 23:45 | |
| *** greghaynes has joined #openstack-infra | 23:49 | |
| *** rbrndt has quit IRC | 23:49 | |
| *** rbrndt has joined #openstack-infra | 23:49 | |
| *** rbrndt has quit IRC | 23:49 | |
| *** thorst has quit IRC | 23:50 | |
| *** ijw has joined #openstack-infra | 23:53 | |
| *** rossella_s has quit IRC | 23:57 | |
| *** rossella_s has joined #openstack-infra | 23:58 | |
| fungi | thanks for the heads up jeblair! | 23:58 |
| *** greghaynes has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!