Monday, 2022-10-10

*** ysandeep|out is now known as ysandeep03:58
ysandeepgood morning ci o/04:21
*** amoralej|off is now known as amoralej06:12
*** ysandeep is now known as ysandeep|afk06:23
marioso/06:48
*** jpena|off is now known as jpena07:07
jm1ysandeep|afk, marios, #oooq: good monday morning :D07:24
marios\o jm1 happy monday ;)07:30
mariosping if you need anything jm1|ruck arxcruz|rover or if you want to give me some task if it gets busy07:31
jm1marios: thanks! let me see what our current status is first. rlandy posted something about a gateblocker..07:33
mariosjm1: yeah looks like undercloud-upgrade was just looking at that 07:34
mariosjm1: https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade 07:34
mariosjm1: hmm looks related to the fix we merged for another cix 07:35
mariosfyi amoralej fix for https://bugs.launchpad.net/bugs/1989795 seems to cause https://bugs.launchpad.net/tripleo/+bug/1992305 07:36
mariosamoralej: Depsolve Error occurred - package python3-sqlalchemy13-1.3.24-2.el9s.x86_64 conflicts with python3-sqlalchemy >= 1.4 provided by python3-sqlalchemy-1.4.39-2.el9s.x86_64 Edit07:36
mariosamoralej: maybe we still need to exclude 1.4? 07:36
mariosdid we revert that? 07:36
marioschecking07:36
amoralejmarios, you removed the exclude?07:36
amoralejmarios, does it happen in periodic?07:37
amoraleji think what we need is to get https://review.rdoproject.org/r/q/topic:sqlalchemy13 promoted07:37
mariosamoralej: we didn't remove it i was just checking https://opendev.org/openstack/tripleo-quickstart/src/commit/dc26b849018e3d5f2ecfe625fca7264877fe75ae/config/release/tripleo-ci/CentOS-9/wallaby.yml#L191 07:37
amoralejthe problem is that you exclude 1.4.3707:37
amoralejbut now, there is 1.4.4007:38
amoralejso it's actually doing nothing07:38
amoralejconflicts with python3-sqlalchemy >= 1.4 provided by python3-sqlalchemy-1.4.40-1.el9s.x86_607:38
amoralejbut anyway, i think https://review.rdoproject.org/r/c/openstack/oslo-db-distgit/+/45589 should fix that issue07:38
amoralejwe need promotions07:38
mariosamoralej: k last wallaby 9 promotion from 06th and looks like those merged 07th 07:39
amoralejyes07:39
jm1marios: anything we should do for amoralej's promotions?07:39
*** ysandeep|afk is now known as ysandeep07:39
mariosjm1: ^^ promotion should help us on that one fyi wallaby907:39
mariosjm1: well chase it/make it a priority depending on how bad the others are looking... 07:39
amoralejmarios, as workaround we may extend the exclude to 1.4*07:40
mariosjm1: k train is not great wallaby8 is ok master not bad so 07:40
mariosamoralej: yeah i was going to ask that too 07:40
mariosamoralej: might be faster so ... 07:40
amoralejbut, if we can get promotions and remove the exclude would be perfect07:40
mariosamoralej: jm1: i'll add that too ^^^ 1.4* now and jm1 if you can try and focus on wallaby9 promotion when you are promotion chasing today? 07:40
amoralejbtw, in periodic jobs07:41
jm1marios: ack, so high prio to c9 wallaby promo 07:41
mariosyeah thank you jm1 07:41
amoralejare you seeing any issue related to sqlalchemy?07:41
mariosamoralej: well we have the workaround there too so.. probably it was hitting there too? https://opendev.org/openstack/tripleo-quickstart/commit/054053061579c626d960b38cd2bb0874de1651bb 07:42
mariosamoralej: python3-sqlalchemy-1.4.*x86_64   <-- should be ok? 07:43
amoraleji'd say so07:43
mariosthx amoralej 07:45
mariosjm1|ruck: arxcruz|rover: meet in 43 OK for you (at 1100 cet)08:17
mariosi'll send invite08:17
mariosjm1|ruck: arxcruz|rover: to go over the cix cards 08:17
jm1marios: for me, yes :D08:18
mariosjm1: k we can start and arxcruz|rover will catch us 08:18
mariosif he is away now08:18
jm1marios: "k train is not great" << what is not so great here? from promotion pov we are good, looks like we have no new content to promote. 08:19
mariosjm1: yeah i just meant it was a few days old but ... no new content is there any known issue with the components? (i.e. is new content blocked?)08:19
jm1marios: nope, no content blocked for train, not in components, not in integration :)08:22
mariosjm1: k weird though i mean 5 days is quite long... i guess ahem... the train is slowing down probably ... 08:23
marios(sorry couldn't help myself)08:23
jm1marios: this sqlalchemy bug is also hitting c9 master. but this should also be fixed when c9 wallaby promoted, right? (since it is doing an upgrade from wallaby to master)08:25
mariosjm1: hmm good point... in fact we may need to expand the temp fix to also have the exclude for master then? /me checking bug again 08:25
jm1marios: imho we could just wait for c9 wallaby promo. only one job is missing, internal kvm and it is running right now08:26
mariosjm1: but to your question no, i think the issue in this case is during the upgrade .. so indeed it goes wallaby to master but i think we are getting it in the 'master' part of the job not he wallaby part of it08:26
mariosie not during the deployment of wallaby, but during the upgrade to master08:27
jm1marios: hmm.. so what to do?08:28
mariosjm1: updating the temp workaroudn to include master cc amoralej ^^ the new bug is hitting the master job i.e. wallaby to master upgrade so we need this in master release file too (this = temp exclude )08:29
mariosjm1: also if we get the wallaby promotion and it is green no problem we abandon the extra patch 08:37
mariosjm1: race! o/08:38
jm1marios: 🤸08:41
arxcruz|roverjm1 marios i'm back 09:13
mariosarxcruz|rover: please join us? 09:13
marioshttps://meet.google.com/yki-daae-pxr09:13
marioshttps://opendev.org/openstack/openstack-tempest-skiplist/src/commit/c542f0e0073d23487a45ec937a3e3a0a79a00950/roles/validate-tempest/vars/tempest_skip.yml#L68609:39
*** ysandeep is now known as ysandeep|lunch09:52
jm1marios, amoralej: we have a new c9 wallaby promotion10:16
amoralejgreat!10:17
amoralejso,  could we check an upgrade job?10:17
jm1amoralej: yes, let me try10:18
mariosjm1: great... i suspect we need it for master too though so may still need the temp fix10:20
mariosjm1: i.e. wallaby promotion would fix the 'older' bug but i don't htink it will fix the new one? 10:21
mariosbut lets check jm1 10:21
amoralejmarios, the new one doesn't need the fix10:22
amoralejit's wallaby only issue10:22
jm1marios: just rechecked c9 master undercloud upgrade job, lets see what happens10:22
jm1arxcruz|rover: ^ used your testproject for that10:23
mariosamoralej: well, you have excluded 1.4 from wallaby, but the new bug from today/gate blocker is hitting undercloud-upgrade *master*10:23
amoralejbut in the wallaby deployment part10:23
amoralejright?10:23
mariosamoralej: no during the package update to master i.e. undercloud-upgrade.log10:23
amoralejlet me double check10:24
mariosamoralej: thats why i think we may need the temp fixor will excludeing during th ewallaby deployment help us on upgrade to master? don't think so ?10:24
amoralejno10:24
amoralejthe issue may be different10:25
amoralejmarios, let me try to reproduce out of the job10:29
amoralejlet me know how the job goes10:29
jm1marios, amoralej: false positive, c9 wallaby still not promoted, sorry :/10:35
amoraleji'm simulating the failed task10:36
amoralejasuming we have a promotion10:36
amoraleji found the issue10:41
amoralejmmm10:41
jm1arxcruz|rover: undercloud deployment on internal kvm job is timing out, any idea what the issue could be? https://sf.hosted.upshift.rdu2.redhat.com/logs/61/416561/4/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby/877ef77/10:41
mariosjm1: k np 10:41
mariosamoralej: can you add info in the launchpad if you found sthing else?10:43
mariosamoralej: jm1: problem with master and https://review.opendev.org/c/openstack/tripleo-quickstart/+/860810 10:43
amoralejmarios, yes, i'm checking how to fix ...10:43
mariosamoralej: jm1: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a6b/860810/2/check/tripleo-ci-centos-9-undercloud-upgrade/a6b9fdb/logs/undercloud/home/zuul/undercloud_upgrade.log   Problem: package python3-oslo-db-12.1.0-0.20220929080853.7f3647b.el9.noarch requires python3-sqlalchemy >= 1.4.010:43
mariosamoralej: ok thanks 10:44
mariosjm1: commented/-1 @ https://review.opendev.org/c/openstack/tripleo-quickstart/+/860810/2#message-ce0249b7d214c9bb26985b5351904f48ec5b88ea 10:44
amoralejshort version is that we need to pass --allowerasing when doing the full update10:46
amoralejmarios, jm1 what version of ansible are we using when running that?10:48
amoralejbtw, adding the exclude to master will not help10:49
mariosamoralej: yeah it did not it caused something else in fact https://review.opendev.org/c/openstack/tripleo-quickstart/+/860810/2#message-ce0249b7d214c9bb26985b5351904f48ec5b88ea 10:50
mariosamoralej: we need 1.4+ there for master10:50
amoralejyes10:51
arxcruz|roverjm1 10:52
arxcruz|roverjm1 i'll be back in 1 hour, lunch and take my daugther 10:53
jm1arxcruz|rover: ack10:54
amoralejmarios, jm1 i just commented in https://bugs.launchpad.net/tripleo/+bug/199230510:54
jm1ysandeep|lunch, Tengu: our c9 wallaby internal kvm job is timing out. i am wondering why it takes more than 1h to run "tripleo_firewall : Manage firewall rules". is this expected? https://sf.hosted.upshift.rdu2.redhat.com/logs/61/416561/4/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby/877ef77/logs/undercloud/home/zuul/standalone_deploy.log10:56
amoraleji guess it's https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_packages/tasks/upgrade.yml ...10:56
amoralejor it's https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/undercloud/undercloud-upgrade.yaml#L141-L146 ?10:58
jm1chandankumar: is there any load test or something running on ibm which might explain why c9 wallaby internal kvm job is failing?11:05
jm1..with timeout11:05
chandankumarjm1: cs9 wallaby internal kvm job, it does not run on IBM cloud11:06
chandankumarinternal must be running on psi11:06
jm1chandankumar: perfect, than it will fail on somethign different :)11:06
jm1chandankumar: ack, thank you!11:08
chandankumarjm1: Do you have the logs11:08
chandankumaror let me check cix if there is any card with running kvm issue11:08
jm1chandankumar: https://sf.hosted.upshift.rdu2.redhat.com/logs/61/416561/4/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby/877ef77/logs/undercloud/home/zuul/standalone_deploy.log11:08
jm1chandankumar: one thing which might be unusual is that "tripleo_firewall : Manage firewall rules" takes more than 1h11:09
chandankumarjm1: https://trello.com/c/R6SuOv6E/2661-cixlp1983718tripleociproa-periodic-master-scen1-standalone-fails-timeout-manage-firewall-rules11:11
Tengujm1: upstream, there was an issue with the DNS iirc11:13
*** ysandeep|lunch is now known as ysandeep11:15
*** dviroel|out is now known as dviroel11:24
dviroelo/11:24
ysandeepakahat, bhagyashris As you have know most about promoter, Do you want to take a look at https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45468 and see if it okay.11:35
bhagyashrisysandeep, ack11:36
dviroelysandeep: i have a comment that needs fix on this patch, finishin my review11:39
ysandeepdviroel, ack11:41
ysandeeppojadhav, hey fyi.. https://review.opendev.org/c/openstack/tripleo-ci/+/860586/3#message-2e1e0b36e08b668a88f12f0f049de5524da341de 11:46
ysandeepdviroel, fixed the typo in rlandy's patch, nice catch!11:53
dviroeltks11:53
jm1Tengu: do you mean the one that chandankumarlinked above your message?12:09
jm1chandankumar: thanks for the link! i am unsure how to handle this undercloud timeout issue on internal kvm job. shall i open a new bug? shall i reuse the one linked in the cix card above? https://bugs.launchpad.net/tripleo/+bug/198371812:12
jm1marios: ^12:12
ysandeepjm1, You can use same bug if its the same issue, I remember I have added a workaround earlier for rdo12:17
ysandeepjm1, https://review.opendev.org/c/openstack/tripleo-ci/+/85475112:18
jm1ysandeep: i dont know if it is the same issue. if >1h for tripleo_firewall is ok, then it might be something else. 12:19
ysandeepI had added steps on bug on how to debug the issue, let me grab the steps12:19
mariosjm1: i always prefer re-using bugs if we have same issue as the context is all in one place 12:20
mariosjm1: but always "it depends" if you can justify/happier with new bug go for it just maybe link between them 12:20
ysandeepjm1, yes just run "time iptables -t filter -L INPUT" and see how long it takes and if /etc/resolv.cong have 127.0.0.1 as first entry.12:21
ysandeepresolv.conf*12:21
* ysandeep checking logs12:21
ysandeephmm, not 127.0.0.1 but we are using some internal dns servers https://sf.hosted.upshift.rdu2.redhat.com/logs/61/416561/4/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby/877ef77/logs/undercloud/etc/resolv.conf 12:24
*** amoralej is now known as amoralej|lunch12:28
Tengujm1: errr yeah. (sorry, was on a call)12:29
jm1amoralej, marios, ysandeep: i am getting lost in all those issues, so let me try to recap: we need a c9 wallaby promo to get the amoralej's sqlalchemy fix. that fix plus amoralej's allowerasing patch will fix our c9 master undercloud-upgrade job, a gate blocker. but atm our c9 wallaby promo is blocked by internal kvm job.12:29
*** dviroel is now known as dviroel|biab12:29
mariosjm1: the sqlalchemy fix is needed but not for the blocker. for that we only waiting for allowerasing12:30
mariosjm1: so the wallaby/9 promotion i think is wanted but not the priority 12:30
mariosjm1: once we get the fix with the 9 promotion it just means we can unpin there https://opendev.org/openstack/tripleo-quickstart/src/commit/dc26b849018e3d5f2ecfe625fca7264877fe75ae/config/release/tripleo-ci/CentOS-9/wallaby.yml#L19112:31
ysandeepFor the "Manage firewall" taking long - I would suggest holding a node and once you are on "Manage Firewall" step.. try "iptables -t filter -L INPUT" and see if it takes time.. We can then figure out if its because of one of internal dns.12:32
*** dasm|off is now known as dasm12:32
dasmo/12:32
mariosgerrit down d/stream? 12:33
ysandeephttps://bugs.launchpad.net/tripleo/+bug/1983718 comment 7, 8 9 will help12:33
mariosThe server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.12:33
ysandeepmarios, checking12:35
jm1ysandeep: looks like there is a dns issue on psi, see google group on "PnT - Infrastructure"12:37
pojadhavysandeep, ack thanks for review. I will fix those review comment.12:38
jm1ysandeep: we have to change our dns servers, dont we?12:38
ysandeepjm1, that's it then, We can wait for dns issue to clear or you are temporary set google dns, As this is rdo job, I think it should work.12:39
jm1ysandeep: dns server has been decomissioned, so i guess we have to change it12:39
jm1ysandeep: i see a looot of references to 10.11.142.1 in our internal repos12:40
jm1ysandeep: i will post patches12:40
jm1Tengu, ysandeep, chandankumar, marios: fyi our internal kvm has wrong dns nameservers which could be causing the undercloud timeout issues. reason is that dns server has been decomissioned. will post patches to change the dns servers12:43
ysandeepjm1, ack, Do you know the new dns servers?12:43
Tengujm1: heh, "nice" :)12:43
jm1ysandeep: yep12:44
ysandeepjm1, great o/12:45
jm1ysandeep: shall we use the the orange ones here? https://source.redhat.com/departments/it/digitalsolutionsdelivery/it-infrastructure/uis/uis_wiki/infoblox_list_of_infoblox_dns_members_to_use_per_geo12:47
* ysandeep checking12:50
mariosthx jm1 12:51
ysandeepjm1, yes looks like it as our jobs run on PSI env12:51
ysandeepbut I wonder if we are getting these dns entries from our Config / Neutron?12:53
ysandeepI think we should also check what neutron dhcp is giving us in this case.12:54
jm1ysandeep: patch per dm12:57
jm1ysandeep: sent you the first of 3 patches12:57
ysandeepNot only internal kvm jobs, but entire downstream jobs are hosted.. see https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-glance/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario004-standalone-glance-rhos-16.2/1246ccb/job-output.txt13:00
mariosysandeep: seems like all internal affected 13:04
mariosysandeep: having trouble opening a repo/pkglist from pkgs.devel.redhat.com/ 13:05
ysandeepthough gerrit worked for me13:05
mariosysandeep: right same gerrit ok 13:06
*** amoralej|lunch is now known as amoralej13:12
*** pojadhav is now known as pojadhav|sick13:13
ysandeepmarios, chandankumar dviroel|biab Could one of you please take a give a second +2 on https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/430747 , needed for downstream.13:16
jm1dasm: good morning :)13:19
mariosack ysandeep 13:19
ysandeepmarios, one more: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/430746, lgtm13:20
dasmjm1: o/13:21
arxcruz|rovermarios jm1 only the upgrade job is failing on master, does it worth to remove it from promotion criteria to promote? we are 4 days behind 13:22
jm1arxcruz|rover: we have fixes for the upgrade job and we might have a solution for failing kvm internal job. maybe we can wait another day?13:23
jm1marios: ^13:23
arxcruz|roverjm1 kvm internal is already being skipped 13:23
arxcruz|roverjm1 marios wallaby cs9 is failing only on fs39 i'm rerunning now, if it's random tempest failure, i'll skip and promote as well 13:24
jm1arxcruz|rover: c9 wallaby internal kvm is in criteria https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config_environments/rdo/CentOS-9/wallaby.yaml#L3713:25
frenzyfridayhey 0/ is the downstream cockpit down?13:25
arxcruz|roverjm1 on wallaby, but not in master 13:25
arxcruz|roverjm1 https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config_environments/rdo/CentOS-9/master.yaml#L5113:26
jm1arxcruz|rover: what i mean is that we just have to merge patches and then we get promotions on c9 master and c9 wallaby13:29
*** dviroel|biab is now known as dviroel13:31
mariosfolks scrum 13:31
jm1arxcruz|rover: ^13:32
mariosamoralej: jm1: looks liek master fix works https://zuul.openstack.org/status#860824 undercloud-upgrade green 13:33
amoralejcool!13:34
arxcruz|roverjm1 i would promote, because we will also need a component promotion once amoralej patch get merged 13:46
jm1arxcruz|rover: iiuc amoralej's patch does not need a component promotion, but maybe amoralej can confirm13:47
arxcruz|roverjm1 it's on tht i think it does 13:47
amoraleji'd say it'd work without promotion13:47
arxcruz|roverbut i might be wrong 13:47
amoralejwell, actually in the gate is using last promotion13:48
arxcruz|roveramoralej ok, i'll run the upgrade with a depends on on your patch 13:48
amoralejso it should be right13:48
jm1amoralej: ok thanks :D13:48
jm1chandankumar: scrum? no? :D13:49
chandankumarjm1: sorry got occupied with something else13:50
jm1chandankumar: ack, thanks for the update :)13:51
Tenguarxcruz|rover: heya! is this one known already? RECHECK13:51
Tenguwoops13:51
Tenguarxcruz|rover: Depsolve Error occurred: \n Problem: package python3-oslo-db-12.1.0-0.20220929080853.7f3647b.el9.noarch requires python3-sqlalchemy >= 1.4.0, but none of the providers can be installed13:51
Tengusorry13:51
arxcruz|roverTengu yes 13:51
Tengu"yay"13:51
Tenguguess it's a blocker.13:52
arxcruz|roverTengu https://review.opendev.org/c/openstack/tripleo-heat-templates/+/860824/ fixes it 13:52
dasmjm1: rcastillo o/ I'm checking our internal ansible playbooks for provisioning servers: eg https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/server_provision/tasks/main.yml#L213:52
arxcruz|roveras an youtuber saying: so go ther and smash that +2 button if you like it and subscribe 13:52
jm1arxcruz|rover, Tengu: depends where he is seeing it ;)13:52
Tengutripleo-ci-centos-9-undercloud-upgrade13:52
Tengujm1, arxcruz|rover -^13:52
jm1arxcruz|rover++ 😂13:52
dasmThey rely on "os_volume" etc. However collections notice they're not anymore supported.13:52
Tenguarxcruz|rover: smashed the +2 already ;)13:53
* dviroel_ primary isp down, thats why i dropped 13:53
dasmjm1: rcastillo Is it the time to move over to collections for openstack?13:53
dasmhere: https://galaxy.ansible.com/openstack/cloud?extIdCarryOver=true&sc_cid=701f2000001OH7YAAW13:53
dasmjm1: rcastillo  if i'm gonna update playbooks, should I be aware of something?13:53
Tenguarxcruz|rover: (re: youtuber - there's also that last part about the bell to get rings or something ;))13:55
rcastillodasm: should be fine switching to the openstack.cloud collection as long as you make sure you install sdk<0.9913:55
dasmrcastillo: something like: "pip install openstacksdk<1.0.0" ?13:56
rcastillono, literally <0.99.013:56
dasmack13:56
dasmrcastillo: based on the documentation, it sounds like "os_volume" et al. are gonna be removed (are removed) for new collections. Is that correct? Do you have any knowledge of that?13:58
rcastillodasm: yes, basically just remove the os_ prefix and you should be good13:59
dasmack13:59
dasmrcastillo++13:59
jm1dasm: its worth to change os_* to openstack.cloud.* because those redirects have been removed in our master branch, hence aoc 2.0.0 will not work our playbooks anylonger13:59
jm1rcastillo: omg i am too slow :D13:59
dasmjm1: ack. that's what i wanted to know.14:00
jm1rcastillo++ :)14:00
rcastillo:)14:00
arxcruz|roverTengu lol yeah, but gerrit already send you emails 14:01
Tengu:]14:06
*** dviroel_ is now known as dviroel14:10
arxcruz|rovermarios can you +w https://review.opendev.org/c/openstack/tripleo-heat-templates/+/860824/ ?14:33
*** ysandeep is now known as ysandeep|out14:35
mariosk arxcruz|rover ideally upgrades folks should sign off on it added them as reviewers in case it is really undesirable but also set workflow ot clear gates 14:39
jm1arxcruz|rover: looks like we are waiting for patches to merge14:56
jm1arxcruz|rover: internal kvm job is still waiting on dns servers patch14:57
jm1arxcruz|rover: so will be eod now14:58
chandankumarsee ya!14:59
mariosjm1|ruck: arxcruz|rover: any problem on cix call ?15:00
jm1marios: nope, arxcruz|rover handled all difficult questions :)15:01
marios:)15:01
* jm1 out for today, have a nice evening :)15:02
rcastillojm1 o/15:02
marioshave a nice evening jm1 o/15:02
dviroelo/15:03
* dviroel lunch15:03
*** dviroel is now known as dviroel|lunch15:03
arxcruz|roverjm1 did I? O.o15:24
marioshave a good one oooci arxcruz|rover need sthing  ? 15:36
arxcruz|rovermarios beer15:37
mariosarxcruz|rover: ack shipping it now 15:37
mariosdrone is taking off15:37
* marios out o/ same again tomorrow yes? 15:37
*** marios is now known as marios|out15:38
arxcruz|roveryes 15:47
arxcruz|roverthanks 15:47
*** amoralej is now known as amoralej|off15:52
*** jpena is now known as jpena|off16:10
*** dviroel|lunch is now known as dviroel|16:21
*** dviroel| is now known as dviroel16:21
* dasm stepping away for some time. bbl17:48
*** dasm is now known as dasm|off17:49
*** dviroel_ is now known as dviroel18:28
*** dviroel is now known as dviroel|biab19:51
*** dasm|off is now known as dasm21:52
rcastillo_ /nick rcastillo23:30
*** rcastillo_ is now known as rcastillo23:30

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!