Tuesday, 2021-05-18

« Monday, 2021-05-17 Index Wednesday, 2021-05-19 »
*** jamesmcarthur has joined #openstack-infra00:41
*** __ministry has joined #openstack-infra01:18
*** ramishra has quit IRC01:19
*** ramishra has joined #openstack-infra01:20
*** gyee has quit IRC01:31
*** jamesmcarthur has quit IRC01:36
*** ramishra has quit IRC02:10
*** ramishra has joined #openstack-infra02:10
*** jamesmcarthur has joined #openstack-infra02:21
*** zxiiro has quit IRC02:56
openstackgerritwushiming proposed openstack/pbr master: Upgrade the pre-commit-hooks version  https://review.opendev.org/c/openstack/pbr/+/79187903:17
*** zbr has quit IRC03:21
*** zbr0 has joined #openstack-infra03:21
*** zbr0 has quit IRC03:30
*** zbr has joined #openstack-infra03:30
*** psachin has joined #openstack-infra03:38
*** markmcclain has quit IRC03:49
*** markmcclain has joined #openstack-infra03:50
*** ykarel has joined #openstack-infra04:25
*** hamalq has quit IRC04:27
*** snapiri has joined #openstack-infra05:22
*** jamesmcarthur has quit IRC05:31
*** jamesmcarthur has joined #openstack-infra05:32
*** vishalmanchanda has joined #openstack-infra05:32
*** jamesmcarthur has quit IRC05:37
*** psachin has quit IRC05:47
*** psachin has joined #openstack-infra05:51
*** jamesmcarthur has joined #openstack-infra06:02
*** sboyron has joined #openstack-infra06:06
*** snapiri has quit IRC06:14
*** slaweq has joined #openstack-infra06:22
*** ralonsoh has joined #openstack-infra06:30
*** jcapitao has joined #openstack-infra06:36
*** dklyle has quit IRC06:41
*** ysandeep|away is now known as ysandeep07:02
*** hashar has joined #openstack-infra07:07
*** hashar is now known as hasharAway07:07
*** andrewbonney has joined #openstack-infra07:13
*** jamesmcarthur has quit IRC07:13
*** nightmare_unreal has joined #openstack-infra07:14
*** rpittau|afk is now known as rpittau07:27
*** jamesmcarthur has joined #openstack-infra07:28
*** auristor has quit IRC07:32
*** hasharAway is now known as hashar07:43
*** tosky has joined #openstack-infra07:46
*** yamamoto has joined #openstack-infra07:49
*** ociuhandu has joined #openstack-infra07:49
*** jpena|off is now known as jpena07:57
*** derekh has joined #openstack-infra07:59
*** lucasagomes has joined #openstack-infra08:05
openstackgerritMatthias Runge proposed openstack/project-config master: Retire panko, python-pankoclient and puppet-panko  https://review.opendev.org/c/openstack/project-config/+/79190508:28
openstackgerritMatthias Runge proposed openstack/project-config master: Retire panko, python-pankoclient and puppet-panko  https://review.opendev.org/c/openstack/project-config/+/79190508:53
*** ociuhandu has quit IRC08:55
*** ociuhandu has joined #openstack-infra08:56
*** ociuhandu has quit IRC08:57
*** ociuhandu has joined #openstack-infra08:57
*** strigazi has joined #openstack-infra08:59
*** gfidente|afk is now known as gfidente09:00
*** tkajinam has quit IRC09:02
*** tkajinam has joined #openstack-infra09:03
openstackgerritMatthias Runge proposed openstack/project-config master: Retire panko, python-pankoclient and puppet-panko  https://review.opendev.org/c/openstack/project-config/+/79190509:05
*** ysandeep is now known as ysandeep|lunch09:10
*** ociuhandu has quit IRC09:56
*** ociuhandu has joined #openstack-infra09:57
*** ociuhandu has quit IRC09:58
*** ociuhandu has joined #openstack-infra09:58
*** jamesmcarthur has quit IRC10:01
*** dtantsur|afk is now known as dtantsur10:14
*** chaconpiza has joined #openstack-infra10:15
*** ysandeep|lunch is now known as ysandeep10:17
*** gtema has joined #openstack-infra10:19
*** yamak16 has quit IRC10:30
*** yamamoto has quit IRC10:34
*** yamamoto has joined #openstack-infra10:47
*** yamamoto has quit IRC10:48
*** ociuhandu has quit IRC10:56
*** gtema has quit IRC10:56
*** ociuhandu has joined #openstack-infra10:57
*** ociuhandu has quit IRC11:00
*** ociuhandu has joined #openstack-infra11:00
*** ociuhandu has quit IRC11:00
*** ociuhandu has joined #openstack-infra11:01
*** ociuhandu has quit IRC11:02
*** ociuhandu has joined #openstack-infra11:02
*** ociuhandu has quit IRC11:02
*** ociuhandu has joined #openstack-infra11:03
*** whoami-rajat has joined #openstack-infra11:06
*** ociuhandu has quit IRC11:08
*** auristor has joined #openstack-infra11:20
*** ociuhandu has joined #openstack-infra11:23
*** ociuhandu has quit IRC11:25
*** ociuhandu has joined #openstack-infra11:25
*** jpena is now known as jpena|lunch11:26
*** yamamoto has joined #openstack-infra11:26
*** dciabrin has quit IRC11:32
*** ociuhandu has quit IRC11:36
*** ociuhandu has joined #openstack-infra11:37
*** carloss has joined #openstack-infra11:41
*** yamamoto has quit IRC11:42
*** ociuhandu has quit IRC11:42
*** ociuhandu has joined #openstack-infra11:45
*** ociuhandu has quit IRC11:51
*** ysandeep is now known as ysandeep|brb11:52
*** jamesmcarthur has joined #openstack-infra11:59
*** ociuhandu has joined #openstack-infra12:04
*** jpena|lunch is now known as jpena12:13
janderssee you tomorrow Ironic o/12:24
*** ociuhandu has quit IRC12:27
*** ociuhandu has joined #openstack-infra12:28
*** tbachman has quit IRC12:29
*** ysandeep|brb is now known as ysandeep12:30
*** tbachman has joined #openstack-infra12:31
*** jcapitao is now known as jcapitao_afk12:32
*** ociuhandu has quit IRC12:33
*** yamamoto has joined #openstack-infra12:34
*** ociuhandu has joined #openstack-infra12:39
*** hashar has quit IRC13:07
*** hashar has joined #openstack-infra13:08
*** dciabrin has joined #openstack-infra13:11
*** nweinber has joined #openstack-infra13:19
*** JanZerebecki[m] has quit IRC13:23
*** JanZerebecki[m] has joined #openstack-infra13:29
*** ociuhandu has quit IRC13:29
*** carloss has quit IRC13:31
openstackgerritRiccardo Pittau proposed openstack/project-config master: Retire sushy-cli - Step 1  https://review.opendev.org/c/openstack/project-config/+/79196313:31
*** gshippey has quit IRC13:31
*** nweinber has quit IRC13:34
*** mnasiadka has quit IRC13:35
*** gagehugo has quit IRC13:35
*** nweinber has joined #openstack-infra13:36
*** mnasiadka has joined #openstack-infra13:37
*** coreycb has quit IRC13:37
*** gshippey has joined #openstack-infra13:37
*** ociuhandu has joined #openstack-infra13:38
*** carloss has joined #openstack-infra13:38
*** coreycb has joined #openstack-infra13:38
*** gagehugo has joined #openstack-infra13:38
*** dchen has quit IRC13:42
openstackgerritRiccardo Pittau proposed openstack/project-config master: Retire sushy-cli - Step 3  https://review.opendev.org/c/openstack/project-config/+/79196713:43
*** bhagyashris has quit IRC13:52
*** bhagyashris has joined #openstack-infra13:54
*** openstack has joined #openstack-infra14:08
*** ChanServ sets mode: +o openstack14:08
fungi"WARNING 2021-05-18T14:00:31 Ping sent at 2021-05-18T13:58:30 not replied to."14:10
*** ysandeep is now known as ysandeep|mtg14:10
fungitosky: it should be back now14:10
fungii'll get statusbot reconnected as well14:10
*** openstackstatus has joined #openstack-infra14:11
*** jamesmcarthur has quit IRC14:11
*** ChanServ sets mode: +v openstackstatus14:11
fungi#status log Restarted "openstack" meetbot and "openstackstatus" statusbot processes after what appears to have been network disruption around 13:58 UTC14:11
openstackstatusfungi: finished logging14:11
toskythanks14:14
fungithanks for the heads up14:15
*** jcapitao_afk is now known as jcapitao14:15
*** dpawlik7 is now known as dpawlik14:19
*** tbachman_ has joined #openstack-infra14:24
*** tbachman has quit IRC14:26
*** tbachman_ is now known as tbachman14:26
*** yamamoto has quit IRC14:34
*** jamesmcarthur has joined #openstack-infra14:35
*** yamamoto has joined #openstack-infra14:36
*** __ministry has quit IRC14:39
*** yamamoto has quit IRC14:40
*** ricolin has quit IRC14:41
*** dklyle has joined #openstack-infra14:45
*** psachin has quit IRC14:46
*** hashar is now known as hasharAway14:48
*** yamamoto has joined #openstack-infra14:52
*** yamamoto has quit IRC14:56
*** jamesdenton has joined #openstack-infra14:59
*** ysandeep|mtg is now known as ysandeep15:01
*** jamesden_ has quit IRC15:02
*** zul has joined #openstack-infra15:22
*** ykarel has quit IRC15:23
*** thiagop has joined #openstack-infra15:29
*** jamesden_ has joined #openstack-infra15:33
*** ociuhandu has quit IRC15:34
*** jamesdenton has quit IRC15:36
*** ociuhandu has joined #openstack-infra15:40
*** ociuhandu has quit IRC15:45
*** hasharAway is now known as hashar15:45
*** ysandeep is now known as ysandeep|dinner15:46
*** tosky_ has joined #openstack-infra15:49
*** tosky has quit IRC15:52
*** ociuhandu has joined #openstack-infra15:55
*** ociuhandu has quit IRC16:00
*** lucasagomes has quit IRC16:02
*** ociuhandu has joined #openstack-infra16:11
*** jamesdenton has joined #openstack-infra16:12
*** ociuhandu has quit IRC16:14
*** ociuhandu has joined #openstack-infra16:14
*** ociuhandu has quit IRC16:14
*** jamesden_ has quit IRC16:15
*** __ministry has joined #openstack-infra16:17
*** __ministry has quit IRC16:17
*** jamesdenton has quit IRC16:18
*** ociuhandu has joined #openstack-infra16:19
*** ociuhandu has quit IRC16:20
*** sreejithp has joined #openstack-infra16:20
*** jamesdenton has joined #openstack-infra16:24
*** ociuhandu has joined #openstack-infra16:30
*** ociuhandu has quit IRC16:36
*** ociuhandu has joined #openstack-infra16:42
*** rpittau is now known as rpittau|afk16:45
sreejithphello all, getting a strange issue on zuul job. this is the error16:49
sreejithphttps://www.irccloud.com/pastebin/7ft6PL5Y/16:49
*** ociuhandu has quit IRC16:49
*** gfidente is now known as gfidente|afk16:49
*** ociuhandu has joined #openstack-infra16:49
fungisreejithp: that paste doesn't load for me, just says "Error fetching paste"16:51
sreejithpoh ok...16:51
fungiokay, i got it to load16:51
fungii guess they're just having intermittent database problems there or something16:51
fungiweird, that site also overrides highlight selection in x1116:52
clarkbcan you link to the job logs? there isn't enough context there to undersatnd what is going on16:52
sreejithphttps://zuul.opendev.org/t/openstack/build/0dd6a54d6e67435f8bc9945768b4222c16:52
sreejithpthis is the link to job16:52
fungihttps://zuul.opendev.org/t/openstack/builds?job_name=airship-hostconfig-operator-test&project=airship/hostconfig-operator16:54
fungilooks like the recent failures for that have been for the same proposed change16:54
fungicould that change be altering the network stack on the job node, or crashing the kernel or sshd maybe?16:55
sreejithpyes.. it was running the gate scripts and suddenly got this message " [Zuul] Log Stream did not terminate"16:55
clarkbsreejithp: yes essentially zuul is saying that it could no longer connect to the test node16:55
clarkbcommon reasons for this are as fungi points out: something breaking networking, something crashing the kernel, sshd being updated, etc16:55
clarkb(it is also possible the problem lies under the job as well) the next thing to check is if the job fails consistently in the same location16:56
fungiyou could potentially also output some additional debugging info to watch in the stream, in case it's not obvious which step is crashing the machine16:57
*** jpena is now known as jpena|off16:58
sreejithpbut the same step passed successfully for a previous commit and there is no change to this script where its failing.17:00
sreejithplet me try adding some more debug to this17:00
clarkbsreejithp: it could be ipv6 specific. the liemstone cloud where that job ran is ipv4 only for external communication which includes connectivity to zuul17:01
clarkbsreejithp: if the job is breaking ipv6 you would only see these errors where ipv6 is relied upon17:01
fungisuccesses and failures were both on ubuntu-bionic nodes, so this doesn't seem to be related to the default nodeset change today17:02
*** thiagop has quit IRC17:02
sreejithpoh ok... also one more thing, i dont  see zuul merge triggering for a commit with workflow  (https://review.opendev.org/c/airship/hostconfig-operator/+/791961)17:04
*** derekh has quit IRC17:05
*** jcapitao has quit IRC17:05
clarkbsreejithp: https://opendev.org/airship/hostconfig-operator/src/branch/master/zuul.d/jobs.yaml#L36-L44 that is your only gate job and it excludes .md modifications17:07
clarkbsreejithp: your change modifies README.md so it does not get gated17:07
clarkbhttps://opendev.org/airship/hostconfig-operator/src/branch/master/zuul.d/projects.yaml#L27-L29 is where the gate jobs list is set17:08
fungiyes, zuul needs to run at least one job for a change17:08
fungiand it needs to be a voting job17:09
fungilooks like it's running airship-hostconfig-operator-test but that job is set non-voting17:09
fungiin check17:09
sreejithpok, let me removed the exclude and try.. thanks17:11
fungiif you really need zuul to not run any jobs but still allow changes to merge, you can add the noop job to check and gate pipelines17:11
fungithat way it's explicit that you intend to merge changes without testing them17:12
clarkbsreejithp: does the hostconfig operator modify the host settings? if so then I strongly suspect that thatis what is going on there17:12
clarkbif it consistently breaks when you start it up on ipv6 instances then I would look at what hostconfig-operator does around network settings17:13
*** ociuhandu has quit IRC17:14
sreejithpno,  currently hostconfig-operator gates are just doing deployment of the operator on a kind cluster.17:15
clarkbright but what does the operator do?17:15
clarkbit is a "hostconfig" operator which makes me suspicious17:15
fungilike, does it alter network settings?17:15
sreejithpthe gates just deploy the crd and bring up the operator.  nothing else is done17:18
sreejithpif you check the previous success job the same step have passed17:18
fungii'm not clear on what a crd is or what an operator is, can you speak in terms of the system itself?17:19
clarkbsreejithp: right I understand that but operators effect changes on the system (they act as an operator) for example rook deploys ceph. What does the hostconfig-operator do when you deploy it17:20
*** ralonsoh has quit IRC17:21
clarkbhttps://opendev.org/airship/hostconfig-operator/src/branch/master/airship-host-config/scripts/ethtool.sh forexample shows it has code to do networking things17:22
clarkbhttps://opendev.org/airship/hostconfig-operator/commit/94ae1039343ba5d5d4da2082dd892949beb361d117:23
clarkbbut I'm not familiar with this tool to know what exactly hte intent is and what all it might be touching17:24
sreejithpwe have to apply a yaml to the operator so that it will perform the operation. eg: https://github.com/airshipit/hostconfig-operator/blob/master/demo_examples/example_ethtool.yaml for using the ethtool script.17:28
sreejithpeven if its performing an operation, it  will be inside the docker containers which kind starts for since kind create k8s cluster inside docker17:28
fungiis that change altering how docker or kubernetes is configured? that can involve changes to the host networking stack as well17:29
fungiespecially if they take over the host's interfaces17:30
clarkbsreejithp: docker containers can use host networking17:30
clarkbsreejithp: the problem here is fungi and I don't know the internal workings of your oeprator. We are asking you to entertain the idea that it could be side effecting change to the netwrok stack that causes this problem because you do understand it17:30
clarkbhttps://opendev.org/airship/hostconfig-operator/src/branch/master/tools/deployment/20_configure_ssh_on_nodes.sh for example17:31
fungiyou could push a non-change for the project as a point of comparison, to see whether it's your change causing this failure scenario or some external change which is now influencing how that job runs17:31
clarkbfungi explicitly called out modifying sshd as one thing that may cause this problem17:32
sreejithpSure... i am checking that17:32
*** andrewbonney has quit IRC17:32
*** nightmare_unreal has quit IRC17:32
*** ysandeep|dinner is now known as ysandeep17:36
clarkbit could also be iptables being impacted? would kind maybe shutdown ipv6 access?17:37
sreejithpnot sure how but just tried a recheck and it worked now... it have passed the step where it failed previously with error17:41
fungiyeah, it does look like the recent success results all ran on ipv4-only providers, while the retry_limit results seem to involve sudden loss of ipv6 connectivity for the job node17:42
clarkbsreejithp: it is running in a different cloud with different networking details17:42
fungiso it's probably not caused by that change, but you've previously managed to merge a change which breaks ipv6 networking17:43
*** dtantsur is now known as dtantsur|afk17:43
clarkbhttps://kind.sigs.k8s.io/docs/user/configuration/#networking maybe you need to tell kind to do a dual stack17:43
clarkbI wonder if the default of ipv4 only enforces that aggressively17:44
*** ysandeep is now known as ysandeep|away17:46
*** zxiiro has joined #openstack-infra17:59
*** hashar has quit IRC18:02
*** hamalq has joined #openstack-infra18:05
*** lpetrut has joined #openstack-infra18:10
*** lpetrut has quit IRC18:10
*** dhill has quit IRC18:12
*** dhill has joined #openstack-infra18:22
*** dwilde has joined #openstack-infra18:39
*** yamamoto has joined #openstack-infra18:43
*** zbr has quit IRC18:48
*** yamamoto has quit IRC18:50
*** zbr has joined #openstack-infra19:00
*** whoami-rajat has quit IRC19:16
*** tosky_ is now known as tosky19:25
*** cloudnull has quit IRC19:48
*** ociuhandu has joined #openstack-infra20:27
*** ociuhandu has quit IRC20:35
*** auristor has quit IRC20:52
*** auristor has joined #openstack-infra20:52
*** slaweq has quit IRC20:57
*** nweinber has quit IRC21:05
*** sboyron has quit IRC21:07
*** cloudnull has joined #openstack-infra21:13
*** auristor has quit IRC21:17
*** auristor has joined #openstack-infra21:17
*** cloudnull has quit IRC21:30
*** dustinc has joined #openstack-infra22:13
*** dave-mccowan has quit IRC22:17
*** dave-mccowan has joined #openstack-infra22:22
*** yamamoto has joined #openstack-infra22:30
*** rcernin has joined #openstack-infra22:36
*** yamamoto has quit IRC22:39
*** yamamoto has joined #openstack-infra22:40
*** rcernin has quit IRC22:44
*** rcernin has joined #openstack-infra22:45
*** tosky has quit IRC23:05
*** JanZerebecki[m] has quit IRC23:27
*** dchen has joined #openstack-infra23:29
*** hamalq has quit IRC23:37
*** dhill has quit IRC23:38
*** JanZerebecki[m] has joined #openstack-infra23:50
« Monday, 2021-05-17 Index Wednesday, 2021-05-19 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!