| *** wolverineav has joined #openstack-infra | 00:01 | |
| *** wolverineav has quit IRC | 00:05 | |
| clarkb | as an observation point the aggregation of check pipeline priority based on gate queue seems to be working well | 00:11 |
|---|---|---|
| clarkb | some of that is likely the slow down we are experiencing | 00:11 |
| clarkb | so probably too early to say it is working great, but it is working great today :) | 00:11 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstack-zuul-jobs master: Update openstackid jobs https://review.openstack.org/625691 | 00:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: WIP Deploy a kubernetes and a gitea https://review.openstack.org/626054 | 00:17 |
| mordred | clarkb: look - a lot of yaml | 00:17 |
| *** rlandy has quit IRC | 00:18 | |
| mordred | clarkb: this is in the commit message - but that is very un-close to being in the form it wants to be in - that's mostly just pushing up the POC working state - cleanup and integration with our existing stuff will ensue | 00:18 |
| *** smarcet has quit IRC | 00:20 | |
| clarkb | mordred: https://review.openstack.org/#/c/626054/1/kubernetes/mariadb-pv.yaml does that force k8s to schedule mariadb to the same instance? | 00:21 |
| clarkb | specifically the label type of local and host path of /mnt/data | 00:21 |
| mordred | clarkb: yah - I believe so - although the plan is to replace that with a galera cluster so that we're resilient to node loss | 00:26 |
| *** mriedem has quit IRC | 00:27 | |
| clarkb | I imagine you'd still run them with that pinning so that if say the DC had a power outage when power came back on things would find their data properly | 00:27 |
| mordred | yeah | 00:28 |
| *** _alastor_ has joined #openstack-infra | 00:29 | |
| *** quiquell|off has quit IRC | 00:29 | |
| mordred | that is, in fact, how the galera kube stuff is set up - basically 3x PV and PVC | 00:29 |
| openstackgerrit | Merged openstack-infra/subunit2sql master: Fix handling of xfail and uxsuccess results in subunit2sql cli https://review.openstack.org/625739 | 00:37 |
| *** sthussey has quit IRC | 00:40 | |
| clarkb | http://logs.openstack.org/32/620632/2/gate/openstack-tox-functional-py35/47bc786/ looks like it might be another ansible double tap maybe? hard to say as that job doesn't get syslog | 00:45 |
| clarkb | ansible times out but ara seems to indicate we ran bindep task successfully | 00:45 |
| pabelanger | https://review.openstack.org/625081 has 2 timeouts too, but it is rax | 00:53 |
| ianw | clarkb: true, but it seems to have been pretty stable before | 00:54 |
| clarkb | pabelanger: ya what I am seeing from logstash is that bhs1 isn't timing out significantly worse than any other cloud except for the zookeeper thing | 00:54 |
| clarkb | ianw: ya makes me wonder if one of the newer ansible updates may be causing it? or making it worse? | 00:54 |
| pabelanger | clarkb: I don't see node_id in inventory yet, guess we haven't restarted nodepool / zuul yet | 00:55 |
| clarkb | ianw: but could also be bhs1 being unhappy again (though globally it doesn't seem to show that) | 00:55 |
| *** hwoarang has quit IRC | 00:55 | |
| clarkb | pabelanger: I think we were waiting on the sdk fixes because otherwise dogpile would break us | 00:55 |
| *** gyee has quit IRC | 00:55 | |
| pabelanger | ah, right | 00:55 |
| *** hwoarang has joined #openstack-infra | 00:57 | |
| clarkb | if we look at the last 12 hours of gate failures there is one in bhs1 and 9 in iad | 00:59 |
| *** armax has joined #openstack-infra | 01:00 | |
| clarkb | if we look at the last 6 hours of all failures. We have ~381 failures. With 66 in bhs1, 65 in ord, 59 in inap, 58 in dfw and iad | 01:00 |
| clarkb | bhs1 is max-servers 150, ord is 195, inap is 190, iad is 145 and dfw is 140 | 01:01 |
| clarkb | so the distributions don't look particularly bad | 01:02 |
| clarkb | (note using the last 6 hours beacuse 12 for all failuers is >500 which makes the kibana tooling weird as its outside the page length of 500 | 01:02 |
| clarkb | bhs1 is maybe a little high on the aggregate total which includes check which may just be noisy | 01:03 |
| *** calbers has quit IRC | 01:03 | |
| clarkb | but looks clean on the gate side of things | 01:03 |
| *** jamesmcarthur has joined #openstack-infra | 01:04 | |
| *** calbers has joined #openstack-infra | 01:07 | |
| *** jklare has quit IRC | 01:08 | |
| *** hwoarang has quit IRC | 01:09 | |
| *** hwoarang has joined #openstack-infra | 01:11 | |
| *** jklare has joined #openstack-infra | 01:14 | |
| *** wolverineav has joined #openstack-infra | 01:18 | |
| *** wolverineav has quit IRC | 01:23 | |
| clarkb | the slow nova metadata api bug shows up in bhs1 and in other regions. I think I'm mostly coming to a conclusion that our software is in general unhappy at times and not unique to bhs1 with some exceptions that we should dig into further (zookeeper) | 01:24 |
| clarkb | but more data necessary | 01:24 |
| *** jamesmcarthur has quit IRC | 01:24 | |
| *** jamesmcarthur has joined #openstack-infra | 01:25 | |
| *** jamesmcarthur has quit IRC | 01:30 | |
| *** _alastor_ has quit IRC | 01:34 | |
| *** jamesmcarthur has joined #openstack-infra | 01:43 | |
| *** hwoarang has quit IRC | 01:43 | |
| *** n00balert has joined #openstack-infra | 01:44 | |
| *** hwoarang has joined #openstack-infra | 01:46 | |
| *** n00balert has quit IRC | 01:46 | |
| *** n00balert has joined #openstack-infra | 01:47 | |
| *** jklare has quit IRC | 01:49 | |
| *** jklare has joined #openstack-infra | 01:58 | |
| *** dayou has quit IRC | 01:59 | |
| *** hwoarang has quit IRC | 02:04 | |
| *** calbers has quit IRC | 02:05 | |
| *** hwoarang has joined #openstack-infra | 02:05 | |
| *** dayou has joined #openstack-infra | 02:06 | |
| *** jamesmcarthur has quit IRC | 02:14 | |
| *** weshay_pto has quit IRC | 02:18 | |
| openstackgerrit | Merged openstack-infra/system-config master: Enable github shared admin account https://review.openstack.org/624531 | 02:29 |
| *** wolverineav has joined #openstack-infra | 02:33 | |
| *** wolverineav has quit IRC | 02:34 | |
| *** wolverin_ has joined #openstack-infra | 02:34 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:39 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] Use bindep for devstack jobs https://review.openstack.org/626068 | 02:39 |
| ianw | frickler: that's the other component of the bindep change I sent into devstack -- I don't think the world is ready for us to install from bindep in devstack automatically, but opt-in like this might work | 02:40 |
| clarkb | ooh | 02:46 |
| clarkb | ianw my nodepool de stack changes from earlier today may inyerest you too | 02:47 |
| clarkb | hoping they are more reliable overall as a reault | 02:47 |
| ianw | clarkb: just looking; tmpfs seems fine ... as i noted it does leave them tmpfs behind | 02:48 |
| clarkb | ya, we do that with the unittest jobs too | 02:50 |
| clarkb | but also leaves zk running so maybe ok? | 02:50 |
| ianw | yeah, we do so much that leaves the system in a state and that certainly isn't idempotent i'm not too worried | 02:51 |
| clarkb | the zk docs do strongly recommend a dedicated device otherwise you can have multisecond delays | 02:52 |
| *** smarcet has joined #openstack-infra | 02:52 | |
| clarkb | other things we can try are longer timeouts or disabling sync to disk but tmpfs should be roughly equivalent | 02:52 |
| *** psachin has joined #openstack-infra | 02:57 | |
| *** mrsoul has quit IRC | 03:00 | |
| *** dave-mccowan has joined #openstack-infra | 03:05 | |
| *** jamesmcarthur has joined #openstack-infra | 03:14 | |
| *** apetrich has quit IRC | 03:15 | |
| *** jamesmcarthur has quit IRC | 03:19 | |
| *** Bhujay has joined #openstack-infra | 03:20 | |
| *** dave-mccowan has quit IRC | 03:24 | |
| *** bobh has joined #openstack-infra | 03:25 | |
| openstackgerrit | Tristan Cacqueray proposed openstack/gertty master: Do not decode failed POST response https://review.openstack.org/626076 | 03:25 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] Add dogpile.cache master to the -src tests https://review.openstack.org/625457 | 03:30 |
| *** yamamoto has quit IRC | 03:34 | |
| *** yamamoto has joined #openstack-infra | 03:34 | |
| *** yamamoto has quit IRC | 03:36 | |
| *** Bhujay has quit IRC | 03:38 | |
| *** hongbin has joined #openstack-infra | 03:51 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 03:54 | |
| *** hwoarang has quit IRC | 03:54 | |
| *** bhavikdbavishi has quit IRC | 03:56 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:56 | |
| *** hwoarang has joined #openstack-infra | 03:56 | |
| *** dklyle has quit IRC | 04:02 | |
| *** david-lyle has joined #openstack-infra | 04:02 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/openstack-zuul-jobs master: Fix name of 'integrated-gate-py35' project template https://review.openstack.org/626078 | 04:04 |
| *** wolverin_ has quit IRC | 04:06 | |
| *** chkumar|out is now known as chandankumar | 04:12 | |
| *** wolverineav has joined #openstack-infra | 04:13 | |
| *** wolverineav has quit IRC | 04:13 | |
| *** armax has quit IRC | 04:13 | |
| *** slaweq has quit IRC | 04:13 | |
| *** udesale has joined #openstack-infra | 04:14 | |
| *** wolverineav has joined #openstack-infra | 04:14 | |
| *** wolverineav has quit IRC | 04:18 | |
| *** ykarel has joined #openstack-infra | 04:22 | |
| *** armax has joined #openstack-infra | 04:25 | |
| *** smarcet has quit IRC | 04:26 | |
| openstackgerrit | Melvin Hillsman proposed openstack-infra/system-config master: Add openci.io to mailman https://review.openstack.org/626084 | 04:29 |
| *** bobh has quit IRC | 04:30 | |
| *** armax has quit IRC | 04:32 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Use renamed template 'integrated-gate-py3' https://review.openstack.org/626087 | 04:33 |
| *** Bhujay has joined #openstack-infra | 04:33 | |
| openstackgerrit | Merged openstack-infra/glean master: fix tox python3 overrides https://review.openstack.org/574332 | 04:34 |
| *** bobh has joined #openstack-infra | 04:43 | |
| *** janki has joined #openstack-infra | 04:48 | |
| openstackgerrit | Merged openstack-infra/glean master: check networkd files https://review.openstack.org/610064 | 04:48 |
| *** hwoarang has quit IRC | 04:50 | |
| *** hwoarang has joined #openstack-infra | 04:52 | |
| *** n00balert has quit IRC | 05:03 | |
| *** evrardjp_ has joined #openstack-infra | 05:04 | |
| *** vabada has quit IRC | 05:05 | |
| *** evrardjp has quit IRC | 05:07 | |
| *** diablo_rojo has quit IRC | 05:10 | |
| *** bobh has quit IRC | 05:22 | |
| *** ykarel has quit IRC | 05:24 | |
| *** ykarel has joined #openstack-infra | 05:36 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Fix container job documentation typo https://review.openstack.org/626093 | 05:38 |
| *** hongbin has quit IRC | 05:40 | |
| *** udesale has quit IRC | 05:45 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Run devstack zookeeper on tmpfs https://review.openstack.org/626038 | 05:46 |
| *** udesale has joined #openstack-infra | 05:49 | |
| *** evrardjp has joined #openstack-infra | 06:01 | |
| *** evrardjp_ has quit IRC | 06:03 | |
| *** wolverineav has joined #openstack-infra | 06:12 | |
| *** gregoryo has joined #openstack-infra | 06:18 | |
| *** wolverineav has quit IRC | 06:21 | |
| *** dims has quit IRC | 06:27 | |
| *** dims has joined #openstack-infra | 06:28 | |
| *** dims has quit IRC | 06:33 | |
| *** dims has joined #openstack-infra | 06:34 | |
| *** mgagne has quit IRC | 06:35 | |
| *** yboaron has joined #openstack-infra | 06:39 | |
| *** mgagne has joined #openstack-infra | 06:40 | |
| *** ianw is now known as ianw_pto | 06:43 | |
| ianw_pto | Not really around till next year ... Merry Christmas & happy new year to all | 06:44 |
| *** diablo_rojo has joined #openstack-infra | 06:52 | |
| frickler | ianw_pto: thx, I'll take a look at your patches later, have a good time | 06:57 |
| AJaeger | ianw_pto: Merry Christmas to you as well! Enjoy the vacation | 07:07 |
| openstackgerrit | Yang Youseok proposed openstack/os-testr master: Add --failing option to run only failed testcases https://review.openstack.org/624893 | 07:08 |
| AJaeger | frickler: want to give ianw_pto a christmas present and +2 his change https://review.openstack.org/#/c/621840 ;) | 07:09 |
| *** quiquell has joined #openstack-infra | 07:13 | |
| *** jamesmcarthur has joined #openstack-infra | 07:15 | |
| *** jamesmcarthur has quit IRC | 07:19 | |
| *** wolverineav has joined #openstack-infra | 07:20 | |
| *** wolverineav has quit IRC | 07:20 | |
| *** wolverineav has joined #openstack-infra | 07:20 | |
| *** dpawlik has joined #openstack-infra | 07:21 | |
| *** dpawlik has quit IRC | 07:25 | |
| *** vabada has joined #openstack-infra | 07:27 | |
| *** imacdonn has quit IRC | 07:29 | |
| *** imacdonn has joined #openstack-infra | 07:29 | |
| *** ykarel is now known as ykarel|lunch | 07:34 | |
| *** dpawlik has joined #openstack-infra | 07:36 | |
| *** dpawlik has quit IRC | 07:36 | |
| *** dpawlik_ has joined #openstack-infra | 07:36 | |
| *** ccamacho has joined #openstack-infra | 07:40 | |
| *** wolverineav has quit IRC | 07:46 | |
| *** slaweq has joined #openstack-infra | 08:00 | |
| *** tosky has joined #openstack-infra | 08:02 | |
| *** pcaruana has joined #openstack-infra | 08:03 | |
| *** rpittau has joined #openstack-infra | 08:04 | |
| *** pgaxatte has joined #openstack-infra | 08:08 | |
| *** markvoelker has joined #openstack-infra | 08:10 | |
| *** bhavikdbavishi has quit IRC | 08:10 | |
| *** jtomasek has joined #openstack-infra | 08:11 | |
| *** ginopc has joined #openstack-infra | 08:14 | |
| *** yboaron has quit IRC | 08:15 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/nodepool master: [wip] Use bindep for devstack jobs https://review.openstack.org/626068 | 08:17 |
| *** Bhujay has quit IRC | 08:19 | |
| *** pcaruana has quit IRC | 08:24 | |
| *** shardy has joined #openstack-infra | 08:25 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:25 | |
| *** gregoryo has quit IRC | 08:31 | |
| *** pcaruana has joined #openstack-infra | 08:33 | |
| *** ykarel|lunch is now known as ykarel | 08:37 | |
| *** pcaruana has quit IRC | 08:41 | |
| *** jogo has joined #openstack-infra | 08:46 | |
| *** jpena|off is now known as jpena | 08:47 | |
| *** e0ne has joined #openstack-infra | 08:47 | |
| *** rcernin has quit IRC | 08:51 | |
| *** alexchadin has joined #openstack-infra | 08:55 | |
| *** gfidente has joined #openstack-infra | 08:59 | |
| *** yboaron has joined #openstack-infra | 09:00 | |
| amorin | hi frickler, fungi, clarkb and others | 09:02 |
| amorin | I can see that BHs1 backend has been enabled back | 09:02 |
| amorin | keep me up to date with the results | 09:03 |
| *** Bhujay has joined #openstack-infra | 09:03 | |
| *** jpich has joined #openstack-infra | 09:04 | |
| *** Bhujay has quit IRC | 09:04 | |
| *** Bhujay has joined #openstack-infra | 09:05 | |
| *** Bhujay has quit IRC | 09:06 | |
| *** Bhujay has joined #openstack-infra | 09:06 | |
| *** Bhujay has quit IRC | 09:07 | |
| *** dkehn has quit IRC | 09:08 | |
| *** Bhujay has joined #openstack-infra | 09:08 | |
| *** Bhujay has quit IRC | 09:09 | |
| *** Bhujay has joined #openstack-infra | 09:09 | |
| *** wolverineav has joined #openstack-infra | 09:10 | |
| *** Bhujay has quit IRC | 09:10 | |
| *** Bhujay has joined #openstack-infra | 09:11 | |
| frickler | amorin: bhs1 seems to be running fine since yesterday evening, failure rates looking good | 09:11 |
| frickler | amorin: I do see very high time-to-ready peaks for gra1, though. probably that might benefit from the fix from bhs1 getting applied there, too? http://grafana.openstack.org/d/rZtIH5Imz/nodepool?orgId=1&from=now-24h&to=now | 09:13 |
| *** wolverineav has quit IRC | 09:15 | |
| amorin | gra1 is affected by the same memory leak yes, | 09:17 |
| amorin | so if it works well on BHS1 | 09:17 |
| amorin | then I can fix GRA1 also | 09:17 |
| *** priteau has joined #openstack-infra | 09:23 | |
| *** Bhujay has quit IRC | 09:24 | |
| *** e0ne has quit IRC | 09:28 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Add count_slot_usage argument for sensitivity https://review.openstack.org/625657 | 09:35 |
| *** derekh has joined #openstack-infra | 09:37 | |
| ianw_pto | fricker: i think http://logs.openstack.org/57/625457/8/check/nodepool-functional-py35-src/ea8e31f/ installed dogpile.cache from source correctly. if it's working, nice template for including github repos in devstack testing | 09:43 |
| *** Bhujay has joined #openstack-infra | 09:48 | |
| *** e0ne has joined #openstack-infra | 09:50 | |
| *** Bhujay has quit IRC | 09:51 | |
| *** bhavikdbavishi has quit IRC | 09:52 | |
| frickler | ianw_pto: it seems to get installed only into the nodepool venv, not globally, not sure if that can lead to inconsistent testing. but the approach itself seems to be working, yes. http://logs.openstack.org/57/625457/8/check/nodepool-functional-py35-src/ea8e31f/job-output.txt.gz#_2018-12-19_08_02_58_062573 | 09:56 |
| ianw_pto | frickler: yeah, this is as intended (or at least, fitting in with the status-quo of what's there, with openstacksdk being installed into the venv) | 10:04 |
| ianw_pto | we were seeing the failure in the job from nodepool-builder as it tried to upload the image; so should this have been around then i think we would have caught the issue | 10:05 |
| ianw_pto | (assuming anyone looked at the test :) | 10:05 |
| *** aojea has joined #openstack-infra | 10:08 | |
| *** diablo_rojo has quit IRC | 10:11 | |
| *** Bhujay has joined #openstack-infra | 10:11 | |
| *** Bhujay has quit IRC | 10:12 | |
| *** Bhujay has joined #openstack-infra | 10:12 | |
| *** Bhujay has quit IRC | 10:13 | |
| *** Bhujay has joined #openstack-infra | 10:14 | |
| frickler | ianw_pto: sadly now clarkb's zk patch seems to break things http://logs.openstack.org/68/626068/2/check/nodepool-functional-py35-src/b6947c4/job-output.txt.gz#_2018-12-19_09_52_55_714283 | 10:17 |
| ianw_pto | frickler: oh, is that a xenail v bionic thing? how weird | 10:18 |
| ianw_pto | oh, it seems to be running before zk is installed | 10:19 |
| frickler | hmm, maybe zookeeper is missing from bindep? | 10:19 |
| ianw_pto | ohh, right that's the bindep change | 10:20 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul master: Adds ServerAliveInterval to ssh_args to prevent frozen connections https://review.openstack.org/626138 | 10:21 |
| ianw_pto | http://logs.openstack.org/68/626068/2/check/nodepool-functional-py35-src/b6947c4/job-output.txt.gz#_2018-12-19_09_42_22_281436 | 10:21 |
| ianw_pto | seems to think everything is installed | 10:21 |
| ianw_pto | bindep.txt:zookeeperd [platform:dpkg test] ... oh, it's only installed under test | 10:21 |
| ianw_pto | I think probably the -bindep argument in devstack is going to need to be maybe "-bindep=profilea,profileb" and we'll have to do some magic to split it up | 10:22 |
| ianw_pto | it could, i suppose, use argparse ... but i think i'd be happy with munging a fixed string in a self contained function | 10:22 |
| ianw_pto | optarg i mean | 10:23 |
| ianw_pto | frickler: feel free to update ;) | 10:23 |
| *** Bhujay has quit IRC | 10:24 | |
| ianw_pto | or we could hard code in "test" as i think that's a common one | 10:24 |
| *** psachin has quit IRC | 10:29 | |
| *** ssbarnea|rover has quit IRC | 10:34 | |
| *** ssbarnea|rover has joined #openstack-infra | 10:35 | |
| *** pbourke has quit IRC | 10:37 | |
| tobias-urdin | ianw_pto: awesome, thanks! | 10:37 |
| *** pbourke has joined #openstack-infra | 10:39 | |
| *** rossella_s has quit IRC | 10:40 | |
| *** ccamacho has quit IRC | 11:05 | |
| *** dpawlik_ has quit IRC | 11:09 | |
| *** dpawlik has joined #openstack-infra | 11:09 | |
| *** e0ne has quit IRC | 11:09 | |
| *** dpawlik has quit IRC | 11:13 | |
| *** udesale has quit IRC | 11:13 | |
| *** dpawlik has joined #openstack-infra | 11:13 | |
| *** ccamacho has joined #openstack-infra | 11:14 | |
| *** dpawlik has quit IRC | 11:14 | |
| *** rossella_s has joined #openstack-infra | 11:14 | |
| *** dpawlik has joined #openstack-infra | 11:14 | |
| *** dpawlik has quit IRC | 11:16 | |
| *** dpawlik has joined #openstack-infra | 11:17 | |
| *** dpawlik has quit IRC | 11:17 | |
| *** dpawlik has joined #openstack-infra | 11:17 | |
| *** Bhujay has joined #openstack-infra | 11:21 | |
| *** jtomasek has quit IRC | 11:37 | |
| *** ginux has joined #openstack-infra | 11:42 | |
| *** jtomasek has joined #openstack-infra | 11:44 | |
| *** ginopc has quit IRC | 11:45 | |
| *** ginux is now known as ginopc | 11:45 | |
| *** rfolco has quit IRC | 11:59 | |
| *** rf0lc0 has joined #openstack-infra | 11:59 | |
| *** e0ne has joined #openstack-infra | 12:01 | |
| *** jonher_ has joined #openstack-infra | 12:05 | |
| *** dpawlik has quit IRC | 12:06 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:06 | |
| *** fresta has joined #openstack-infra | 12:06 | |
| *** jonher has quit IRC | 12:08 | |
| *** jonher_ is now known as jonher | 12:08 | |
| *** fresta- has quit IRC | 12:08 | |
| *** markmcd has quit IRC | 12:11 | |
| *** jtomasek has quit IRC | 12:11 | |
| *** rpittau is now known as rpittau|lunch | 12:19 | |
| *** tpsilva has joined #openstack-infra | 12:26 | |
| *** jpena is now known as jpena|lunch | 12:29 | |
| *** hogepodge has quit IRC | 12:29 | |
| *** sparkycollier has quit IRC | 12:30 | |
| *** hogepodge has joined #openstack-infra | 12:30 | |
| *** sparkycollier has joined #openstack-infra | 12:30 | |
| *** markmcd has joined #openstack-infra | 12:32 | |
| *** gfidente has quit IRC | 12:32 | |
| *** janki has quit IRC | 12:32 | |
| *** gfidente has joined #openstack-infra | 12:33 | |
| *** rh-jelabarre has joined #openstack-infra | 12:40 | |
| *** bhavikdbavishi has quit IRC | 12:42 | |
| *** wolverineav has joined #openstack-infra | 12:47 | |
| *** dpawlik has joined #openstack-infra | 12:50 | |
| *** wolverineav has quit IRC | 12:51 | |
| *** jtomasek has joined #openstack-infra | 12:57 | |
| *** jpena|lunch is now known as jpena | 13:01 | |
| *** rlandy has joined #openstack-infra | 13:03 | |
| *** ykarel is now known as ykarel|away | 13:04 | |
| *** bobh has joined #openstack-infra | 13:04 | |
| *** mriedem has joined #openstack-infra | 13:06 | |
| *** dave-mccowan has joined #openstack-infra | 13:08 | |
| *** ykarel|away has quit IRC | 13:12 | |
| *** jtomasek has quit IRC | 13:13 | |
| *** trown|outtypewww is now known as trown | 13:13 | |
| *** jtomasek has joined #openstack-infra | 13:13 | |
| *** rpittau|lunch is now known as rpittau | 13:19 | |
| *** markvoelker has quit IRC | 13:24 | |
| *** janki has joined #openstack-infra | 13:26 | |
| *** jamesmcarthur has joined #openstack-infra | 13:27 | |
| *** jcoufal has joined #openstack-infra | 13:33 | |
| *** ykarel|away has joined #openstack-infra | 13:34 | |
| *** jamesmcarthur has quit IRC | 13:37 | |
| *** e0ne has quit IRC | 13:43 | |
| *** kgiusti has joined #openstack-infra | 13:47 | |
| *** jamesmcarthur has joined #openstack-infra | 13:48 | |
| *** dkehn has joined #openstack-infra | 13:49 | |
| *** boden has joined #openstack-infra | 13:50 | |
| *** sthussey has joined #openstack-infra | 13:56 | |
| *** ykarel|away is now known as ykarel | 14:00 | |
| *** e0ne has joined #openstack-infra | 14:02 | |
| *** Bhujay has quit IRC | 14:02 | |
| *** Bhujay has joined #openstack-infra | 14:02 | |
| *** Bhujay has quit IRC | 14:03 | |
| *** Bhujay has joined #openstack-infra | 14:04 | |
| *** Bhujay has quit IRC | 14:05 | |
| *** Bhujay has joined #openstack-infra | 14:05 | |
| *** Bhujay has quit IRC | 14:06 | |
| *** Bhujay has joined #openstack-infra | 14:07 | |
| *** Bhujay has quit IRC | 14:12 | |
| *** eharney has joined #openstack-infra | 14:15 | |
| *** bobh has quit IRC | 14:25 | |
| *** ssbarnea|rover has quit IRC | 14:40 | |
| *** ssbarnea has joined #openstack-infra | 14:40 | |
| *** bhavikdbavishi has joined #openstack-infra | 14:42 | |
| *** smarcet has joined #openstack-infra | 14:45 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Consolidate openstack-meeting channels https://review.openstack.org/626182 | 14:50 |
| *** udesale has joined #openstack-infra | 14:50 | |
| *** jonher_ has joined #openstack-infra | 14:56 | |
| *** fresta_ has joined #openstack-infra | 14:57 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/system-config master: Consolidate openstack-meeting channels https://review.openstack.org/626185 | 14:58 |
| *** jonher- has joined #openstack-infra | 14:59 | |
| *** jonher has quit IRC | 14:59 | |
| *** jonher- is now known as jonher | 14:59 | |
| *** fresta has quit IRC | 14:59 | |
| *** fresta has joined #openstack-infra | 14:59 | |
| *** fresta_ has quit IRC | 15:01 | |
| *** jonher_ has quit IRC | 15:02 | |
| *** dpawlik has quit IRC | 15:04 | |
| *** dpawlik has joined #openstack-infra | 15:05 | |
| *** dpawlik has quit IRC | 15:06 | |
| *** dpawlik has joined #openstack-infra | 15:06 | |
| *** jonher_ has joined #openstack-infra | 15:07 | |
| *** fresta_ has joined #openstack-infra | 15:08 | |
| *** rlandy is now known as rlandy|rover | 15:08 | |
| *** fresta has quit IRC | 15:10 | |
| *** jonher has quit IRC | 15:10 | |
| *** jonher_ is now known as jonher | 15:10 | |
| *** dpawlik has quit IRC | 15:11 | |
| *** alexchadin has quit IRC | 15:11 | |
| *** armax has joined #openstack-infra | 15:14 | |
| *** bhavikdbavishi has quit IRC | 15:16 | |
| *** dpawlik has joined #openstack-infra | 15:23 | |
| *** dpawlik has quit IRC | 15:27 | |
| *** quiquell is now known as quiquell|off | 15:29 | |
| *** woojay has joined #openstack-infra | 15:30 | |
| *** ccamacho has quit IRC | 15:35 | |
| *** Bhujay has joined #openstack-infra | 15:57 | |
| *** Bhujay has quit IRC | 15:58 | |
| *** Bhujay has joined #openstack-infra | 15:58 | |
| *** Bhujay has quit IRC | 15:59 | |
| *** Bhujay has joined #openstack-infra | 16:00 | |
| *** ccamacho has joined #openstack-infra | 16:01 | |
| *** Bhujay has quit IRC | 16:01 | |
| *** Bhujay has joined #openstack-infra | 16:01 | |
| *** Bhujay has quit IRC | 16:02 | |
| *** Bhujay has joined #openstack-infra | 16:03 | |
| *** Bhujay has quit IRC | 16:04 | |
| *** Bhujay has joined #openstack-infra | 16:04 | |
| *** Bhujay has quit IRC | 16:05 | |
| *** Bhujay has joined #openstack-infra | 16:06 | |
| *** bobh has joined #openstack-infra | 16:06 | |
| *** bobh has quit IRC | 16:06 | |
| *** Bhujay has quit IRC | 16:07 | |
| *** Bhujay has joined #openstack-infra | 16:07 | |
| *** bobh has joined #openstack-infra | 16:08 | |
| *** Bhujay has quit IRC | 16:08 | |
| *** Bhujay has joined #openstack-infra | 16:09 | |
| *** Bhujay has quit IRC | 16:10 | |
| *** _alastor_ has joined #openstack-infra | 16:10 | |
| *** Bhujay has joined #openstack-infra | 16:10 | |
| *** yboaron_ has joined #openstack-infra | 16:11 | |
| *** Bhujay has quit IRC | 16:11 | |
| *** Bhujay has joined #openstack-infra | 16:12 | |
| *** Bhujay has quit IRC | 16:13 | |
| *** Bhujay has joined #openstack-infra | 16:13 | |
| *** yboaron has quit IRC | 16:14 | |
| *** Bhujay has quit IRC | 16:14 | |
| *** Bhujay has joined #openstack-infra | 16:15 | |
| *** Bhujay has quit IRC | 16:16 | |
| *** Bhujay has joined #openstack-infra | 16:16 | |
| *** Bhujay has quit IRC | 16:17 | |
| *** Bhujay has joined #openstack-infra | 16:18 | |
| *** Bhujay has quit IRC | 16:19 | |
| *** ykarel is now known as ykarel|away | 16:19 | |
| *** Bhujay has joined #openstack-infra | 16:19 | |
| *** Bhujay has quit IRC | 16:20 | |
| *** Bhujay has joined #openstack-infra | 16:21 | |
| *** Bhujay has quit IRC | 16:22 | |
| *** Bhujay has joined #openstack-infra | 16:22 | |
| *** wolverineav has joined #openstack-infra | 16:23 | |
| *** Bhujay has quit IRC | 16:23 | |
| *** gfidente is now known as gfidente|afk | 16:24 | |
| *** Bhujay has joined #openstack-infra | 16:24 | |
| *** eharney has quit IRC | 16:24 | |
| *** Bhujay has quit IRC | 16:25 | |
| *** Bhujay has joined #openstack-infra | 16:25 | |
| *** dpawlik has joined #openstack-infra | 16:26 | |
| *** Bhujay has quit IRC | 16:26 | |
| *** Bhujay has joined #openstack-infra | 16:27 | |
| *** wolverineav has quit IRC | 16:27 | |
| *** Bhujay has quit IRC | 16:28 | |
| *** Bhujay has joined #openstack-infra | 16:28 | |
| *** Bhujay has quit IRC | 16:29 | |
| ssbarnea | clarkb: fungi : i am wondering what is the impact on CI of wide changes like: https://review.openstack.org/#/q/owner:jacky06+status:open | 16:30 |
| *** Bhujay has joined #openstack-infra | 16:30 | |
| *** dpawlik has quit IRC | 16:30 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add unmanaged VM to nodepool functional tests https://review.openstack.org/626357 | 16:31 |
| *** Bhujay has quit IRC | 16:31 | |
| *** janki has quit IRC | 16:31 | |
| *** gyee has joined #openstack-infra | 16:31 | |
| *** Bhujay has joined #openstack-infra | 16:31 | |
| *** Bhujay has quit IRC | 16:32 | |
| *** Bhujay has joined #openstack-infra | 16:33 | |
| *** Bhujay has quit IRC | 16:34 | |
| clarkb | ssbarnea: it will vary on the number and type of tests that those projects run | 16:34 |
| *** Bhujay has joined #openstack-infra | 16:34 | |
| clarkb | generally we discourage people from pushing those changes unless there is agreemwnt beforehand | 16:34 |
| *** udesale has quit IRC | 16:35 | |
| *** Bhujay has quit IRC | 16:35 | |
| *** Bhujay has joined #openstack-infra | 16:36 | |
| fungi | however we don't have an easy way to prevent them (or even know in advance they're going to happen) | 16:36 |
| *** Bhujay has quit IRC | 16:37 | |
| *** Bhujay has joined #openstack-infra | 16:37 | |
| ssbarnea | i asked because is not the first time when I see some similar scripted CRs. i was curious, as something like this could easily become a DDOS. | 16:38 |
| *** Bhujay has quit IRC | 16:38 | |
| *** jtomasek has quit IRC | 16:39 | |
| *** Bhujay has joined #openstack-infra | 16:39 | |
| boden | hi, simple question I hope... will the changes to make lower-constraints use py36 (https://review.openstack.org/#/c/623229/) also apply to stable branch builds (e.g. stable/queens)? | 16:40 |
| *** Bhujay has quit IRC | 16:40 | |
| *** Bhujay has joined #openstack-infra | 16:40 | |
| *** jistr has quit IRC | 16:41 | |
| *** Bhujay has quit IRC | 16:41 | |
| *** jistr has joined #openstack-infra | 16:42 | |
| *** Bhujay has joined #openstack-infra | 16:42 | |
| *** dpawlik has joined #openstack-infra | 16:42 | |
| clarkb | boden based on the branch matcher excluding everything but driverfixes: yes | 16:43 |
| boden | clarkb thank you | 16:43 |
| *** Bhujay has quit IRC | 16:43 | |
| *** Bhujay has joined #openstack-infra | 16:43 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add unmanaged vm to nodepool functional tests https://review.openstack.org/626357 | 16:44 |
| *** Bhujay has quit IRC | 16:44 | |
| *** Bhujay has joined #openstack-infra | 16:45 | |
| *** Bhujay has quit IRC | 16:46 | |
| *** jistr_ has joined #openstack-infra | 16:46 | |
| *** dpawlik has quit IRC | 16:46 | |
| *** Bhujay has joined #openstack-infra | 16:46 | |
| *** jistr has quit IRC | 16:47 | |
| *** Bhujay has quit IRC | 16:47 | |
| *** Bhujay has joined #openstack-infra | 16:48 | |
| *** Bhujay has quit IRC | 16:49 | |
| *** Bhujay has joined #openstack-infra | 16:49 | |
| *** chandankumar is now known as chkumar|vanished | 16:50 | |
| *** Bhujay has quit IRC | 16:50 | |
| *** Bhujay has joined #openstack-infra | 16:51 | |
| *** jistr_ has quit IRC | 16:51 | |
| *** jistr has joined #openstack-infra | 16:52 | |
| *** Bhujay has quit IRC | 16:52 | |
| *** Bhujay has joined #openstack-infra | 16:52 | |
| *** Bhujay has quit IRC | 16:53 | |
| *** Bhujay has joined #openstack-infra | 16:54 | |
| *** yboaron_ has quit IRC | 16:55 | |
| *** Bhujay has quit IRC | 16:55 | |
| *** Bhujay has joined #openstack-infra | 16:55 | |
| *** Bhujay has quit IRC | 16:56 | |
| *** Bhujay has joined #openstack-infra | 16:57 | |
| *** dpawlik has joined #openstack-infra | 16:58 | |
| *** Bhujay has quit IRC | 16:58 | |
| *** Bhujay has joined #openstack-infra | 16:58 | |
| fungi | ssbarnea: we have the ability to dequeue them and ban the proposing account in gerrit if it becomes an issue | 16:59 |
| *** Bhujay has quit IRC | 16:59 | |
| *** Bhujay has joined #openstack-infra | 17:00 | |
| fungi | i'd rather not engineer more draconian prevention than is warranted given we have means to recover quickly if needed | 17:00 |
| *** Bhujay has quit IRC | 17:01 | |
| fungi | now that the `zuul dequeue` rpc client subcommand is available it's even easier. in olden days we would dump a copy of the pipeline contents, stop the scheduler, trim out the offending changes from the dump, then start the schedule and reenqueue the rest | 17:01 |
| *** Bhujay has joined #openstack-infra | 17:01 | |
| *** Bhujay has quit IRC | 17:02 | |
| *** dpawlik has quit IRC | 17:03 | |
| *** Bhujay has joined #openstack-infra | 17:03 | |
| *** Bhujay has quit IRC | 17:04 | |
| *** Bhujay has joined #openstack-infra | 17:04 | |
| *** e0ne has quit IRC | 17:05 | |
| *** Bhujay has quit IRC | 17:05 | |
| *** Bhujay has joined #openstack-infra | 17:06 | |
| *** rpittau has quit IRC | 17:06 | |
| *** Bhujay has quit IRC | 17:07 | |
| *** Bhujay has joined #openstack-infra | 17:07 | |
| *** Bhujay has quit IRC | 17:08 | |
| *** Bhujay has joined #openstack-infra | 17:09 | |
| *** Bhujay has quit IRC | 17:10 | |
| *** Bhujay has joined #openstack-infra | 17:10 | |
| *** jpich has quit IRC | 17:11 | |
| *** Bhujay has quit IRC | 17:11 | |
| *** Bhujay has joined #openstack-infra | 17:12 | |
| *** Bhujay has quit IRC | 17:13 | |
| *** Bhujay has joined #openstack-infra | 17:13 | |
| *** Bhujay has quit IRC | 17:14 | |
| *** Bhujay has joined #openstack-infra | 17:15 | |
| *** Bhujay has quit IRC | 17:16 | |
| *** Bhujay has joined #openstack-infra | 17:16 | |
| *** pgaxatte has quit IRC | 17:17 | |
| *** Bhujay has quit IRC | 17:17 | |
| *** Bhujay has joined #openstack-infra | 17:18 | |
| *** e0ne has joined #openstack-infra | 17:19 | |
| *** Bhujay has quit IRC | 17:19 | |
| *** Bhujay has joined #openstack-infra | 17:19 | |
| *** Bhujay has quit IRC | 17:20 | |
| *** Bhujay has joined #openstack-infra | 17:21 | |
| *** Bhujay has quit IRC | 17:22 | |
| *** Bhujay has joined #openstack-infra | 17:22 | |
| *** Bhujay has quit IRC | 17:23 | |
| *** Bhujay has joined #openstack-infra | 17:24 | |
| *** e0ne has quit IRC | 17:24 | |
| *** Bhujay has quit IRC | 17:25 | |
| *** Bhujay has joined #openstack-infra | 17:25 | |
| *** Bhujay has quit IRC | 17:26 | |
| ssbarnea | clarkb: fungi mriedem: how about splitting timeouts into POST and non-POST - https://review.openstack.org/#/c/625573/ -- already included changes to avoid overlapping matches. | 17:27 |
| *** Bhujay has joined #openstack-infra | 17:27 | |
| *** Bhujay has quit IRC | 17:28 | |
| *** Bhujay has joined #openstack-infra | 17:28 | |
| mriedem | ssbarnea: will take a look in a bit | 17:28 |
| fungi | seems a reasonable choice, though i haven't looked at the implementation, i can see where tracking them separately could be useful | 17:29 |
| *** Bhujay has quit IRC | 17:29 | |
| *** ginopc has quit IRC | 17:30 | |
| *** Bhujay has joined #openstack-infra | 17:30 | |
| *** yumapath_ has joined #openstack-infra | 17:31 | |
| *** Bhujay has quit IRC | 17:31 | |
| yumapath_ | hi folks after the zuul base jobs mandated the log-inventory role | 17:31 |
| yumapath_ | we are facing the following issue | 17:31 |
| *** Bhujay has joined #openstack-infra | 17:31 | |
| yumapath_ | 2018-12-18 02:38:02.152982 | primary | "msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'inventory_file'\n\nThe error appears to have been in '/tmp/fe6c300e779b4833a900c6c9c710ba5b/trusted/project_1/git.zuul-ci.org/zuul-jobs/roles/log-inventory/tasks/main.yaml': line 11, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe o | 17:32 |
| yumapath_ | has anyone seen this before? | 17:32 |
| fungi | yumapath_: is this appearing in a job running from zuul.openstack.org or in a zuul deployment you manage? | 17:32 |
| *** Bhujay has quit IRC | 17:32 | |
| yumapath_ | in the zuul deployment that we manage | 17:33 |
| yumapath_ | until that checkin happened things were fine with us | 17:33 |
| *** Bhujay has joined #openstack-infra | 17:33 | |
| yumapath_ | after that checkin got mandated 10 days ago | 17:33 |
| yumapath_ | things are starting to fail | 17:33 |
| fungi | yumapath_: probably better to have this discussion in #zuul in that case, as the zuul-jobs repository is part of the zuul project | 17:33 |
| ssbarnea | fungi: on e-q i would really like to be able have single graph with multiple graphs inside, but current design does not allow this. it would better use the screen space, useful for sibling queries, like that ones. | 17:33 |
| yumapath_ | with the above error | 17:33 |
| yumapath_ | ok thanks | 17:34 |
| *** Bhujay has quit IRC | 17:34 | |
| *** yumapath_ has quit IRC | 17:34 | |
| *** Bhujay has joined #openstack-infra | 17:34 | |
| *** Bhujay has quit IRC | 17:35 | |
| *** Bhujay has joined #openstack-infra | 17:36 | |
| openstackgerrit | Merged openstack-infra/zuul master: Adds ServerAliveInterval to ssh_args to prevent frozen connections https://review.openstack.org/626138 | 17:36 |
| *** Bhujay has quit IRC | 17:37 | |
| *** Bhujay has joined #openstack-infra | 17:37 | |
| *** Bhujay has quit IRC | 17:38 | |
| *** Bhujay has joined #openstack-infra | 17:39 | |
| *** Bhujay has quit IRC | 17:40 | |
| *** Bhujay has joined #openstack-infra | 17:40 | |
| mnaser | hi infra-root: i'd like to request a hold on a job if possible? it uses a multinode nodeset so not sure if that might be different | 17:41 |
| *** Bhujay has quit IRC | 17:41 | |
| mnaser | https://review.openstack.org/#/c/625448/ -- magnum-kubernetes-conformance under openstack/magnum | 17:42 |
| fungi | mnaser: it's handled the same. i'll add that | 17:42 |
| *** Bhujay has joined #openstack-infra | 17:42 | |
| mnaser | fungi: great, thank you. i'll throw a recheck when ready, but this is super super exciting stuff | 17:42 |
| mnaser | deploy multinode openstack, create magnum cluster (using nested virt), run k8s conformance tests on that | 17:43 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Trim devstack services used in testing https://review.openstack.org/626044 | 17:43 |
| *** Bhujay has quit IRC | 17:43 | |
| fungi | i've just now added `sudo zuul autohold --tenant openstack --project openstack/magnum --job magnum-kubernetes-conformance --change 625448 --reason "fungi/mnaser debugging" --count 1` | 17:43 |
| *** Bhujay has joined #openstack-infra | 17:43 | |
| mnaser | fungi: thank you :D | 17:44 |
| *** derekh has quit IRC | 17:44 | |
| *** Bhujay has quit IRC | 17:44 | |
| *** Bhujay has joined #openstack-infra | 17:45 | |
| *** Bhujay has quit IRC | 17:46 | |
| *** Bhujay has joined #openstack-infra | 17:46 | |
| fungi | any time! | 17:47 |
| *** Bhujay has quit IRC | 17:47 | |
| *** dpawlik has joined #openstack-infra | 17:48 | |
| *** Bhujay has joined #openstack-infra | 17:48 | |
| *** Bhujay has quit IRC | 17:49 | |
| *** Bhujay has joined #openstack-infra | 17:49 | |
| *** Bhujay has quit IRC | 17:50 | |
| *** Bhujay has joined #openstack-infra | 17:51 | |
| *** aojea has quit IRC | 17:52 | |
| *** Bhujay has quit IRC | 17:52 | |
| *** Bhujay has joined #openstack-infra | 17:52 | |
| *** dpawlik has quit IRC | 17:53 | |
| *** Bhujay has quit IRC | 17:53 | |
| *** Bhujay has joined #openstack-infra | 17:54 | |
| *** jpena is now known as jpena|off | 17:54 | |
| clarkb | ssbarnea: related to that I was thinking it would be good if we could somehow represent the deduping of bugs from the same job. Some times there is a chain reaction where one bug causes another bug and job fails and we record that as two failures in different graphs | 17:55 |
| *** Bhujay has quit IRC | 17:55 | |
| clarkb | ssbarnea: but the reality is it is one failed job that exhibits signs of multiple different issues and seeing that visually somehow would be helpful to learn what issues are related | 17:55 |
| *** Bhujay has joined #openstack-infra | 17:55 | |
| *** trown is now known as trown|lunch | 17:56 | |
| *** Bhujay has quit IRC | 17:56 | |
| ssbarnea | clarkb: true. quite often. maybe we should be able to define a dependency graph which we can use for sorting the most likely root reason. | 17:57 |
| *** Bhujay has joined #openstack-infra | 17:57 | |
| ssbarnea | still, in its current form using e-r for this is far too hard, especially due to the slow review process. | 17:57 |
| *** Bhujay has quit IRC | 17:58 | |
| clarkb | well its also holiday period so things will slow down across the map | 17:58 |
| ssbarnea | in the past i used bug trackers with custom-fileds for "log-regex-match". but LP does not have such option. | 17:58 |
| *** Bhujay has joined #openstack-infra | 17:58 | |
| ssbarnea | btw, i will be offline for the next 4 days (PTO). | 17:59 |
| *** Bhujay has quit IRC | 17:59 | |
| *** Bhujay has joined #openstack-infra | 18:00 | |
| *** rascasoft has joined #openstack-infra | 18:02 | |
| *** dpawlik has joined #openstack-infra | 18:04 | |
| *** Bhujay has quit IRC | 18:05 | |
| clarkb | elastic recheck is basically that match system. It isn't regexes because lucene and goes through code review because bad queries can easily tank the cluster unfortunately | 18:08 |
| clarkb | we have too much data :/ | 18:08 |
| *** dpawlik has quit IRC | 18:09 | |
| *** wolverineav has joined #openstack-infra | 18:12 | |
| *** wolverineav has quit IRC | 18:12 | |
| *** wolverineav has joined #openstack-infra | 18:12 | |
| *** wolverineav has quit IRC | 18:13 | |
| *** anteaya has joined #openstack-infra | 18:17 | |
| *** wolverineav has joined #openstack-infra | 18:19 | |
| *** shardy has quit IRC | 18:32 | |
| *** e0ne has joined #openstack-infra | 18:33 | |
| *** tobiash has quit IRC | 18:35 | |
| *** smarcet has quit IRC | 18:36 | |
| *** e0ne has quit IRC | 18:40 | |
| *** ykarel|away has quit IRC | 18:43 | |
| *** tobiash has joined #openstack-infra | 18:44 | |
| fungi | anything special i should know about the process for launching new servers now, after not having done one for a few months? i'll need to update the static inventory afterward presumably... anything else? | 18:46 |
| fungi | should we start using opendev.org for server names now that we have it working? | 18:46 |
| fungi | about to build the replacement storyboard-dev | 18:47 |
| *** tobiash has quit IRC | 18:47 | |
| *** tobiash has joined #openstack-infra | 18:48 | |
| clarkb | when I built new etherpad servers there was something that kept me from using opendev.org and I don't remember what it was. Oh I know. The puppet inventory | 18:50 |
| clarkb | er manifest | 18:50 |
| clarkb | fungi: ^ we probably want to do a wholesale update of the site.pp to accept openstack.org or opendev.org | 18:50 |
| clarkb | until then you probably want openstack.org for now | 18:50 |
| fungi | k | 18:51 |
| *** Vadmacs has joined #openstack-infra | 18:51 | |
| fungi | i can take a stab at a site.pp update for that, but we'll likely want to merge it rapidly lest it be delayed by perpetual merge conflicts | 18:51 |
| corvus | i'm performing destructive dev work on the gitea poc; expect it to be non-functional for a while | 18:55 |
| *** jamesmcarthur has quit IRC | 19:01 | |
| *** trown|lunch is now known as trown | 19:06 | |
| *** diablo_rojo has joined #openstack-infra | 19:06 | |
| fungi | this is where trying to match up regexes in the global site manifest and globs in the inventory groups gets fun | 19:11 |
| *** yboaron_ has joined #openstack-infra | 19:12 | |
| corvus | clarkb, fungi, mordred: http://38.108.68.96/zuul/zuul now with rst rendering support (via pandoc) | 19:12 |
| *** jamesmcarthur has joined #openstack-infra | 19:13 | |
| corvus | [also, that wasn't non-functional nearly as long as i thought it was going to be] | 19:13 |
| anteaya | very pretty | 19:14 |
| *** jamesmcarthur has quit IRC | 19:17 | |
| mnaser | pretty | 19:17 |
| mnaser | also wow it loads so quickly | 19:17 |
| mnaser | surely it must be running on a super badass cloud | 19:18 |
| mnaser | (sorry, i had to, back to regular programming) | 19:18 |
| corvus | mnaser: :) the amazing thing is how many layers of ceph it's going through :) | 19:18 |
| mnaser | hopefully if we get manila up, you can just get cephfs right away | 19:18 |
| mnaser | but.. we'll see about that | 19:18 |
| mnaser | i suspect it wont be an easy migration either | 19:19 |
| corvus | i'd love that :) | 19:19 |
| clarkb | ya ceph running on ceph is maybe almost as bad as qemu running on kvm :P | 19:19 |
| corvus | clarkb: it doesn't *seem* bad so far :) | 19:20 |
| clarkb | corvus: mostly just thinking about all the replicas you'll end up writing | 19:20 |
| pabelanger | cool, looks really neat | 19:20 |
| mnaser | clarkb: that same cloud is running qemu on qemu for kata and magnum | 19:20 |
| corvus | yeah, i mean, it's not ideal, it just doesn't seem to be a dealbreaker like qemu-on-kvm. it seems to be a real thing. | 19:20 |
| tbarron | mnaser: please let us know if you have any issues deploying manila | 19:20 |
| clarkb | corvus: ya | 19:21 |
| clarkb | corvus: I wonder if we could/should run without replicas on the inner ceph knowing the outer ceph is doing that for us | 19:21 |
| mnaser | tbarron: absolutely. it's a little later on the roadmap, we're hacking on magnum things right now to get those right | 19:21 |
| mnaser | clarkb: i'd recommend against that | 19:21 |
| clarkb | but then there are other points of failure to worry about | 19:21 |
| mnaser | for availablity reasons | 19:21 |
| clarkb | mnaser: ya | 19:21 |
| mnaser | while data won't disappear, it can become unavailable | 19:21 |
| corvus | yeah... *if* we could convince rook to use k8s volumes for the osds, that might be possible | 19:22 |
| tbarron | mnaser: are you planning just cephfs native behind manila or also cephfs with nfs? | 19:22 |
| corvus | even then, i'm not sure that covers all the failure points | 19:22 |
| tbarron | mnaser: the latter has some advantages for public or enterprise clouds with untrusted compute instances https://www.slideshare.net/Inktank_Ceph/ceph-day-berlin-practical-cephfs-and-nfs-using-openstack-manila | 19:23 |
| mnaser | tbarron: right now, i'd love to do cephfs native but there is a lot of wild things that can cause. nfs introduces a really nice layer of abstraction *but* we create a bottleneck | 19:23 |
| mnaser | but.. we use like, 40g nics across the entire cloud so maybe it's not also really a wild bottleneck. | 19:23 |
| tbarron | mnaser: and for container workloads your container-host many be trusted enuf | 19:24 |
| *** sshnaidm|off has quit IRC | 19:24 | |
| mnaser | tbarron: well none of our workloads are ever trusted, so that's part of why the nfs might be the best alternative overall | 19:24 |
| *** dpawlik has joined #openstack-infra | 19:25 | |
| tbarron | mnaser: we're working on nfs-gateway-per-tenant which will scale out the load so that performance bottleneck putting nfs gateway in front of cephfs isn't too bad, also building more async processing into ganesha gateway and smarter itneraction with cephfs for delegations, etc. | 19:26 |
| clarkb | corvus: I imagine that part of the issue with that is osds want to talk to raw devices but the volumes abstraction gives you a filesystem mount? | 19:26 |
| mnaser | infra-root: there's a hold on 625448 for me :) can i get https://github.com/mnaser.keys in there ? | 19:26 |
| mnaser | tbarron: that's awesome. and they sit outside the project of the user? | 19:27 |
| *** sshnaidm has joined #openstack-infra | 19:27 | |
| corvus | clarkb: honestly, i haven't dug into it yet. | 19:27 |
| clarkb | corvus: or at least that is true if using bluestore. The older slower mechanism that uses xfs? might not care so much | 19:28 |
| corvus | i mean, i know what happens now, i just don't know what the other options are. | 19:28 |
| tbarron | mnaser: yes, the gateways are under cloud admin control. We're trying to move them and the ceph daemons "outside" openstack, like an appliance back end would be | 19:28 |
| fungi | mnaser: sure, just a sec | 19:28 |
| tbarron | mnaser: orchestrating them with k8s we think | 19:28 |
| corvus | clarkb: right. currently we're using bluestore on a device. we can configure it to use the filesystem too. i just don't know if rook is set up to know that part of the filesystem could be a k8s volume. | 19:28 |
| mnaser | tbarron: i like that architecture :D exciting stuff | 19:28 |
| tbarron | mnaser: and use kuryr for the datapath back in to openstack/neutron | 19:29 |
| mnaser | once i get conformance tests going, one of the next things im looking at is native kuryr support in magnum | 19:29 |
| mnaser | tbarron: so you can just ask deployers to have magnum and you can rely on it to deploy :D | 19:29 |
| *** dpawlik has quit IRC | 19:30 | |
| tbarron | mnaser: cool, container workloads are TBH probably my main driver for manila relevance | 19:30 |
| tbarron | mnaser: manila never had tight integration with nova, which was a disadvantage for traditional OpenStack | 19:30 |
| clarkb | tbarron: what integration does there need to be? | 19:31 |
| clarkb | tbarron: you'd nfs mount from inside the instance right? | 19:31 |
| fungi | mnaser: root@38.108.68.90 and root@38.108.68.22 | 19:31 |
| clarkb | (so nova would never really know or care about it) | 19:31 |
| mnaser | fungi: thank you! | 19:31 |
| tbarron | mnaser: but the loose coupling -- we deliver storage over a network rather than through a hypervisor -- is an advantage for servicng storage to container orchestrator hosts | 19:31 |
| clarkb | containers are different in that mounting filesystems is typically managed by the orchestration system to avoid exposing the host filesystem | 19:31 |
| *** jamesmcarthur has joined #openstack-infra | 19:32 | |
| tbarron | clarkb: we don't have anything like cinder-attach from nova | 19:32 |
| clarkb | tbarron: right but its completely unnecessary | 19:32 |
| tbarron | clarkb: some of our potential consumers would like it, they want storage just to "show up" | 19:32 |
| clarkb | (and as a user that is one of the big reasons I'd prefer the fs workflow over the cinder volume one, we spend a lot of time working around that with cinder) | 19:33 |
| clarkb | for example the 16 device limit on Xen VMs | 19:33 |
| clarkb | and the need to check if the volume should be formated before using it | 19:33 |
| clarkb | (hope you don't get that wrong) | 19:33 |
| clarkb | tbarron: there is a halfway point using cloud inits mount facilities iirc | 19:34 |
| tbarron | clarkb: I actually agree with you but some potential customers of manila don't. They want a filesystem to show up, mounted, after issuing an OpenStack command | 19:34 |
| clarkb | tbarron: you could set metadata on the instance that says "mount this manila fs" | 19:34 |
| tbarron | clarkb: kinda before my time in manila but back in mitaka manila proposed putting info in nova metadata to facilitate an automount and nova pushed back | 19:35 |
| tbarron | clarkb: given the history of cinder attach it may have been a prudent reaction | 19:35 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add unmanaged vm to nodepool functional tests https://review.openstack.org/626357 | 19:35 |
| clarkb | tbarron: ya this is user driven instead via user supplied metadata | 19:35 |
| tbarron | clarkb: right | 19:35 |
| clarkb | I see the argument, I just think in practice it causes more problems than it solves. Another with cinder is the volume stuck in an attached state and you really need ti mounted elsewhere :) | 19:36 |
| clarkb | with nfs you'd just mount it | 19:36 |
| tbarron | clarkb: mnaser: but if the wind keeps blowing in the direction that application users want containers with RWX storage then the OpenStack users for manila will be hosts (Vms or baremetal) doing the mountinng of shares and that tight nova integration isn't a big deal | 19:37 |
| fungi | well, you _can_ set up automounting of hotplugged block devices when they appear too, but yes they need to be preformatted | 19:37 |
| tbarron | The OpenStack user is a CO (k8s, whatever) admin, likely with its own automation -- like CSI plugin | 19:38 |
| *** gouthamr_ is now known as gouthamr | 19:38 | |
| clarkb | tbarron: ya and in that case the system mounts for you because it must for security reasons | 19:38 |
| clarkb | (otherwise the host kernel risks exposing itself to the container and being eaten/modified/deleted) | 19:38 |
| tbarron | clarkb: yeah, system mounts and then exposes to application user via bind mount | 19:39 |
| clarkb | I do think finding the balance where you have the option to do it that way or not has been an interesting struggle in general for openstack. I find that often times when we veered towards hand holding as a service we've failed to provide base functionality or expected reliability | 19:40 |
| *** e0ne has joined #openstack-infra | 19:40 | |
| clarkb | but when we don't people complain that $task is too difficult | 19:40 |
| *** dpawlik has joined #openstack-infra | 19:41 | |
| clarkb | probably the lesson there is do our best to provide the hand holding but give power users and out if they need or want it | 19:42 |
| tbarron | clarkb: +1 | 19:43 |
| *** yboaron_ has quit IRC | 19:43 | |
| fungi | huh, i guess the old review.openstack.org is still sitting in a shutoff state, never deleted | 19:45 |
| fungi | and puppetmaster.openstack.org is still up and running? | 19:45 |
| fungi | are we due to clean both of those up? | 19:45 |
| fungi | happy to snapshot and delete them | 19:45 |
| clarkb | fungi: ++ | 19:46 |
| *** dpawlik has quit IRC | 19:46 | |
| fungi | came up as i was going through inventory/groups.yaml and manifests/site.pp | 19:46 |
| *** tobiash has quit IRC | 19:53 | |
| *** tobiash has joined #openstack-infra | 19:58 | |
| *** wolverineav has quit IRC | 19:58 | |
| fungi | infra-root: we also have a 1tb snapshot of the bandersnatch volume from august 2017. safe to delete? | 19:59 |
| *** jamesmcarthur has quit IRC | 19:59 | |
| clarkb | considering we deleted pypi mirror I think so | 20:00 |
| fungi | my suspicion as well | 20:00 |
| mordred | ++ | 20:02 |
| openstackgerrit | Merged openstack-infra/zuul master: web: update status page layout based on screen size https://review.openstack.org/622010 | 20:02 |
| fungi | #status log deleted 1tb "before-run" snapshot of "bandersnatch-temp" volume in rackspace dfw | 20:03 |
| openstackstatus | fungi: finished logging | 20:03 |
| fungi | the volume that was snapshotted from also still exists in an unattached state | 20:03 |
| *** david-lyle has quit IRC | 20:04 | |
| fungi | shall i delete that as well i suppose? | 20:04 |
| clarkb | hrm that I dont know, is it not part of afs? | 20:06 |
| mordred | auristor: if you happen to be around and know the answer - I saw some mention on the internets about a kernel/kafs volume driver for kubernetes, but I can't find any references to the code existing anywhere. is that an auristorfs private thing? or am I just bad at looking for things? | 20:08 |
| fungi | clarkb: it's called "bandersnatch-temp" so i think it was part of the transition | 20:08 |
| clarkb | mordred: wouldnt it be have k8s mount /afs for you and hostsets up /afs? | 20:10 |
| clarkb | using local | 20:10 |
| *** jcoufal has quit IRC | 20:12 | |
| fungi | #status log deleted 1tb "bandersnatch-temp" volume in rackspace dfw | 20:13 |
| openstackstatus | fungi: finished logging | 20:13 |
| fungi | we have 5 other unattached volumes in dfw as well | 20:14 |
| auristor | mordred: auristor demo'd some interactions please see the asciinema links in this talk https://indico.cern.ch/event/676324/contributions/2992389/ | 20:14 |
| mordred | clarkb: yah - that's what a flexvolume driver can do for you - basically have the node mount /afs and then propagate that mount into the containers | 20:14 |
| mordred | clarkb: https://indico.cern.ch/event/676324/contributions/2992389/attachments/1651067/2641278/AuriStor-HEPiX-Spring-2018-AFS-Containers.pdf slide 13 | 20:14 |
| mordred | auristor: yeah - that was the thing I was looking at :) | 20:14 |
| *** markvoelker has joined #openstack-infra | 20:14 | |
| *** dklyle has joined #openstack-infra | 20:15 | |
| fungi | clarkb: of those, two are 1tb each (one for nb04 which isn't in that provider at all, and one for the old nodepool.o.o server) | 20:15 |
| auristor | I will return in a few hours. | 20:15 |
| fungi | clarkb: the others look like leftover volumes from server replacements where we didn't reuse their existing volumes and replaced them instead | 20:17 |
| fungi | eavesdrop.openstack.org/main01 (there is an attached main02 of the same size and type), mirror01.dfw.rax.openstack.org/main01 and main02 (there is a volume attached to mirror02 which is larger than their combined sizes) | 20:18 |
| fungi | as all are unattached i expect they're safe for deletion | 20:18 |
| *** markvoelker has quit IRC | 20:19 | |
| clarkb | the nodepool ones should be as it was always ephemeral data | 20:20 |
| fungi | #status log deleted unattached eavesdrop.openstack.org/main01 (50gb ssd), mirror01.dfw.rax.openstack.org/main01 (100gb), mirror01.dfw.rax.openstack.org/main02 (100gb), nb04.openstack.org/main01 (1tb) and nodepool.openstack.org/main01 (1tb) volumes in rackspace dfw | 20:23 |
| openstackstatus | fungi: finished logging | 20:23 |
| *** e0ne has quit IRC | 20:23 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add unmanaged vm to nodepool functional tests https://review.openstack.org/626357 | 20:23 |
| *** Vadmacs has quit IRC | 20:26 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add gitea dockerfile https://review.openstack.org/626387 | 20:29 |
| *** wolverineav has joined #openstack-infra | 20:30 | |
| *** wolverineav has quit IRC | 20:34 | |
| fungi | clarkb: i'm noticing we still have a bunch of host globs in the futureparser hostgroup which have an explicit [0-9] when the servers are lacking them (lists.o.o but also others). do you know of existing changes to correct those? | 20:37 |
| fungi | i'm guessing those will result in the futureparser changes taking effect on those servers for real | 20:37 |
| *** dkehn has quit IRC | 20:39 | |
| clarkb | ya I havent pushed fixes for them | 20:39 |
| clarkb | in part because it would require monitoring | 20:39 |
| *** priteau has quit IRC | 20:40 | |
| fungi | okay, i'll try to avoid accidentally correcting them in my giant change | 20:40 |
| *** wolverineav has joined #openstack-infra | 20:45 | |
| *** xgerman has joined #openstack-infra | 20:46 | |
| xgerman | o/ | 20:46 |
| xgerman | I am trying to copy a log file (see https://review.openstack.org/#/c/624835/10/zuul.d/jobs.yaml) but somehow they don’t copy… does this only work on folders? | 20:47 |
| *** wolverineav has quit IRC | 20:50 | |
| corvus | #status log converted opendevorg user on dockerhub to an organization owned by openstackzuul | 20:52 |
| openstackstatus | corvus: finished logging | 20:52 |
| *** bobh has quit IRC | 20:53 | |
| *** dkehn has joined #openstack-infra | 20:55 | |
| *** wolverineav has joined #openstack-infra | 20:57 | |
| fungi | #status log deleted old puppetmaster.openstack.org and review.openstack.org servers in rackspace dfw after creating final snapshots | 20:57 |
| openstackstatus | fungi: finished logging | 20:57 |
| *** ssbarnea has quit IRC | 21:00 | |
| *** tosky has quit IRC | 21:02 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Remove deleted review and puppetmaster servers https://review.openstack.org/626391 | 21:03 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Use regexes for servers in global site manifest https://review.openstack.org/626392 | 21:03 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Use domain-generic regexes/globs for hostnames https://review.openstack.org/626393 | 21:03 |
| fungi | clarkb: do you think the end result of that ^ would have solved the issue you ran into? | 21:03 |
| *** smarcet has joined #openstack-infra | 21:03 | |
| *** wolverineav has quit IRC | 21:09 | |
| *** wolverineav has joined #openstack-infra | 21:09 | |
| *** wolverineav has quit IRC | 21:14 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add gitea dockerfile https://review.openstack.org/626387 | 21:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add gitea dockerfile https://review.openstack.org/626387 | 21:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add gitea dockerfile https://review.openstack.org/626387 | 21:22 |
| *** smarcet has quit IRC | 21:23 | |
| *** kgiusti has left #openstack-infra | 21:28 | |
| *** bobh has joined #openstack-infra | 21:29 | |
| *** bobh has quit IRC | 21:34 | |
| rm_work | which channel is diskimage-builder? | 21:35 |
| rm_work | oh derp nm i already have it | 21:35 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Use domain-generic regexes/globs for hostnames https://review.openstack.org/626393 | 21:37 |
| clarkb | fungi yup Im in a really bad spot to help monitor that go in though | 21:40 |
| fungi | okay, i'm around and can keep tabs on it | 21:40 |
| *** dpawlik has joined #openstack-infra | 21:42 | |
| fungi | infra-root: any other available eyes on that stack of changes (626391, 626392, 626393)? happy to approve and monitor them myself. i expect they'll end up in rebase purgatory if we don't merge them fairly quickly | 21:43 |
| *** wolverineav has joined #openstack-infra | 21:46 | |
| *** dpawlik has quit IRC | 21:46 | |
| *** trown is now known as trown|outtypewww | 21:47 | |
| clarkb | can even split the groups and site.pp changes into two changes if you want to make them smaller chunks | 21:49 |
| *** wolverineav has quit IRC | 21:51 | |
| fungi | well, i did break up the switch to regular expressions from explicit hostnames and the domain generalization | 21:53 |
| fungi | in hopes that would make it easier to separate out what was being changed | 21:53 |
| fungi | i suppose with the merger of 624531 yesterday it's safe to remove myself from https://github.com/orgs/openstack/teams/infra-core now? | 21:54 |
| *** wolverineav has joined #openstack-infra | 21:55 | |
| *** wolverineav has quit IRC | 21:55 | |
| *** wolverineav has joined #openstack-infra | 21:56 | |
| clarkb | fungi: I don't know if ianw got around to adding the account as admin on our orgs | 21:57 |
| clarkb | fungi: but if that has been done then yes should be able to remove your account | 21:57 |
| fungi | what was the account name? there's a "OpenStack Infrastructure" (openstackadmin) account in that group | 21:58 |
| *** dpawlik has joined #openstack-infra | 21:58 | |
| clarkb | I think that is correct, but double check in the passwords file? | 21:58 |
| fungi | though i note a lot of our org owner teams haven't been converted to differently-named teams with owner roles yet | 21:58 |
| fungi | which i expect means it hasn't been added to them | 21:59 |
| *** dpawlik has quit IRC | 22:02 | |
| fungi | clarkb: yep, info on how to generate a 2fa token on bridge.o.o for that shared gh account is definitely present in the list | 22:03 |
| fungi | `oathtool --totp -b ...` | 22:04 |
| fungi | along with passphrase | 22:04 |
| clarkb | ya so the biggest thing is just making sure the account has all the relevant perms your account has/had | 22:08 |
| *** calbers has joined #openstack-infra | 22:12 | |
| fungi | confirmed i'm able to authenticate with the totp token and am now accepting all the orgs i invited it into (after i converted their owner roles to infra-core teams) | 22:13 |
| clarkb | I'm not sure I grok that last bit, but I'm guessing its a github thing | 22:13 |
| *** slaweq has quit IRC | 22:13 | |
| fungi | yeah, there used to be an owners list per org | 22:15 |
| *** markvoelker has joined #openstack-infra | 22:15 | |
| fungi | now "owner" is a role you apply to one or more arbitrary teams | 22:15 |
| fungi | so if you haven't gone to the membership page for an org since that transition, they prompt you to convert it to a team and give it a name | 22:16 |
| clarkb | ah | 22:16 |
| fungi | pretty much all our orgs besides "openstack" were in that situation, so i've corrected it and named the teams for them consistently | 22:17 |
| fungi | and then invited the openstackadmin account to all of them | 22:17 |
| fungi | and then switched to that account authenticated via the instructions in the file, and accepted all the invitations | 22:17 |
| fungi | #status log added openstackadmin account to the following additional github orgs: gtest-org, openstack-ci, openstack-dev, stackforge, openstack-infra, openstack-attic, stackforge-attic | 22:19 |
| openstackstatus | fungi: finished logging | 22:19 |
| *** rcernin has joined #openstack-infra | 22:19 | |
| *** bobh has joined #openstack-infra | 22:19 | |
| fungi | and now i'm signed out and going back in with my personal account to leave the infra-core team in each of those orgs | 22:20 |
| fungi | and i'm setting openstackadmin as a maintainer of each of them before i leave too | 22:22 |
| *** bobh has quit IRC | 22:23 | |
| *** boden has quit IRC | 22:25 | |
| *** diablo_rojo has quit IRC | 22:29 | |
| *** diablo_rojo has joined #openstack-infra | 22:31 | |
| fungi | and signed back in as openstackadmin again and confirmed it's seeing itself as a maintainer in the infra-core team of all 8 orgs now | 22:32 |
| fungi | so i'm fully extracted and have the same access via the shared account to all the orgs my personal account formerly was able to maintain | 22:33 |
| *** rcernin has quit IRC | 22:36 | |
| *** rcernin has joined #openstack-infra | 22:37 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add gitea dockerfile https://review.openstack.org/626387 | 22:40 |
| corvus | mordred: ^ i have done a crazy thing with multi-stage builds there; if you can take a moment to look over it before i go too far down the rabbit hole, that would be great :) | 22:42 |
| corvus | (essentially, i'm going for 2 images so we can have 2 containers in the gitea pod, and i've factored out the dockerfile to build the common stuff, then make the 2 images on top of that; i'm using the --target option of docker build to extract those two images from a single dockerfile) | 22:44 |
| corvus | also, fun fact i learned -- you can totally do this in a docker-compose file :) | 22:45 |
| mordred | corvus: looking | 22:45 |
| xgerman | still wondering why https://review.openstack.org/#/c/624835/10/zuul.d/jobs.yaml doesn’t work as expected to copy the ‘octavia-amphora.log' | 22:45 |
| corvus | i guess to be more docker i should capitalize "AS" | 22:47 |
| clarkb | corvus: I left a couple notes that I thought mgiht help | 22:47 |
| clarkb | xgerman: which job that ran are children of that job? | 22:48 |
| clarkb | xgerman: it would probably be helpful if your jobs that use the legacy jobs as parent were also labeled as legacy jobs | 22:48 |
| xgerman | yes: job is http://logs.openstack.org/35/624835/10/check/octavia-v2-dsvm-scenario/2993dba/ | 22:49 |
| xgerman | I went through ara but couldn’t find it in the list of things to copy | 22:50 |
| corvus | clarkb, xgerman: yeah, i think that only works for the zuulv3 native devstack job, not jobs that inherit from legacy-dsvm-base | 22:50 |
| clarkb | xgerman: corvus http://logs.openstack.org/35/624835/10/check/octavia-v1-dsvm-scenario/0f8da2c/logs/octavia-amphora.log.gz it seems to have worked? | 22:51 |
| clarkb | http://logs.openstack.org/35/624835/10/check/octavia-v2-dsvm-scenario/2993dba/controller/logs/ seems to be a non legacy job | 22:51 |
| clarkb | so ya updating labels to make it clear which is which is likely the fix here :) | 22:52 |
| xgerman | ah, must have gotten confused then | 22:52 |
| mordred | corvus, clarkb: I also left some comments, but will now read scrollback | 22:52 |
| mordred | none of this scrollback is about the image builds :) | 22:52 |
| *** tpsilva has quit IRC | 22:53 | |
| xgerman | thanks a dozen clarkb | 22:54 |
| corvus | clarkb, mordred: replied | 22:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add gitea dockerfile https://review.openstack.org/626387 | 22:54 |
| *** bobh has joined #openstack-infra | 22:55 | |
| mordred | corvus: nod. that's helpful | 22:56 |
| clarkb | ya the comment makes it all clear to me | 22:56 |
| clarkb | I think my only concern now would be if having multiple images out of one dockerfile is considered a bad thing | 22:56 |
| clarkb | otherwise lgtm | 22:56 |
| corvus | yeah, it's a little twisted, but i'm trying to reuse as much of the upstream stuff as possible | 22:57 |
| clarkb | (I don't know if that is a bad thing) | 22:57 |
| corvus | i consider the fact that docker-compose handles this use case natively to be a strong indication this is the right track | 22:57 |
| mordred | yeah | 22:57 |
| *** rcernin has quit IRC | 22:57 | |
| *** rcernin has joined #openstack-infra | 22:58 | |
| mordred | corvus: if you want to micro-optimize image-size more - you could put your rm /var/lib/apt/lists/* (and maybe add /var/cache/apt/* to that) after each apt-get install line with an && - and also do an update at the front - at the expense of image build time of course | 22:58 |
| *** rcernin has quit IRC | 22:59 | |
| mordred | corvus: the things being removed in the final images will still get pulled as part of the previous layer commits | 22:59 |
| corvus | mordred: oh. hrm. i had intended to keep the index around until the end, but i forgot about layers | 23:00 |
| *** bobh has quit IRC | 23:00 | |
| clarkb | and if you want to really optomize there is a super minimal golang image base that is just like libc and your binary (I want to say you have to find your own ca chains :/) | 23:00 |
| corvus | so i really do need to remove the lists and apt-get update each time, huh? | 23:00 |
| mordred | corvus: if you want to optimize in that manner, yeah | 23:02 |
| clarkb | corvus: ya otherwise you make a layer with that content and you pull it in | 23:02 |
| clarkb | aiui this is one of the things that makes kolla images large. They haven't optomized those intermediate build things out | 23:03 |
| mordred | that said - it's only 3 apt-get interactions, so it's not TERRIBLE | 23:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add gitea dockerfile https://review.openstack.org/626387 | 23:04 |
| clarkb | fwiw your gitea build may not need to clean unless the dirs you copy from are cleaned by that | 23:04 |
| clarkb | (since that image is a throwaway | 23:04 |
| mordred | clarkb: yah - that's a good point | 23:04 |
| corvus | yeah, i left that one alone <shrug> | 23:04 |
| clarkb | that would be a runtime optomization meh | 23:05 |
| clarkb | save a second or two | 23:05 |
| corvus | but i added a clean to base and added updates to gitea and openssh | 23:05 |
| *** rcernin has joined #openstack-infra | 23:07 | |
| *** rascasoft has quit IRC | 23:12 | |
| *** bobh has joined #openstack-infra | 23:15 | |
| *** rlandy|rover is now known as rlandy|rover|bbl | 23:24 | |
| *** jamesmcarthur has joined #openstack-infra | 23:24 | |
| *** jamesmcarthur has quit IRC | 23:29 | |
| *** rcernin has quit IRC | 23:38 | |
| *** bobh has quit IRC | 23:38 | |
| *** rcernin has joined #openstack-infra | 23:41 | |
| mordred | corvus: I left one more set of comments - since the patch needs a respin anyway | 23:44 |
| *** wolverineav has quit IRC | 23:45 | |
| corvus | mordred: ack willdo | 23:45 |
| *** wolverineav has joined #openstack-infra | 23:46 | |
| *** rascasoft has joined #openstack-infra | 23:48 | |
| *** wolverineav has quit IRC | 23:50 | |
| *** rascasoft has quit IRC | 23:54 | |
| *** wolverineav has joined #openstack-infra | 23:55 | |
| *** tosky has joined #openstack-infra | 23:56 | |
| *** slaweq has joined #openstack-infra | 23:58 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add gitea dockerfile https://review.openstack.org/626387 | 23:58 |
| corvus | turns out the sshd container needs the gitea binary too, so the images aren't that different after all | 23:58 |
| corvus | but those images are working now, and are what the poc is currently running | 23:59 |
| *** dpawlik has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!