[00:01:07] aha [00:01:12] I have confirmed the presence of the same problem in labtest [00:01:47] Krenair: you have access to designate right? if not i'm happy to paste the results here [00:01:58] Not in real-labs [00:02:01] aah [00:02:01] Well [00:02:07] https://www.irccloud.com/pastebin/9ZgMvsTl/ [00:02:20] I have the novaadmin pass as a production deployer [00:02:44] which means I can hit the openstack APIs, just not the underlying databases [00:02:50] alright [00:03:37] So here's the extent of the problem in labtest [00:04:40] https://phabricator.wikimedia.org/P4078 [00:05:01] (note the different domain_id) [00:07:21] all those are deleted instances in labtest with existing ptr records [00:07:54] I imagine prod-labs' one is many times larger given contintcloud [00:07:58] right [00:08:58] https://phabricator.wikimedia.org/diffusion/GSNF/browse/master/nova_fixed_multi/base.py [00:09:09] this is a code that handles creation of the labs internal dns data [00:09:27] and, theoretically, deletion [00:13:55] Krenair: brb [00:19:36] Krenair: ok back, looking at this code [00:20:08] nothing stands out at me as wrong here [00:20:36] Think I'm going to live hack it on labtest to be noisy about what it's up to [00:21:41] sure, let me know what you see [00:29:08] I created and deleted one and it was cleaned up fine.. [00:29:18] i would like to know, out of curiosity, if I can install jekyll on a tool account [00:29:28] madhuvishy, ^ [00:30:31] Krenair: ha, clearly that doesn't always happen - just makes it harder to debug :| [00:30:56] madhuvishy, select max(nova.instances.deleted_at) from records join recordsets on records.recordset_id = recordsets.id left join nova.instances on replace(nova.instances.uuid, '-', '') = records.managed_resource_id where records.domain_id = '8d114f3c815b466cbdd49b91f704ea60' and recordsets.name like '%.10.in-addr.arpa.' and recordsets.type = 'PTR' and nova.instances.deleted_at is not null; [00:31:39] it's probably gonna take a while [00:31:51] running [00:32:24] a while? that's worrying [00:32:48] any idea how many rows nova.instances has? [00:33:27] CristianCantoro: sure. `gem install jekyll --user-install` should work. [00:33:56] Krenair: 236862 [00:34:11] that's not a small number of instances [00:34:55] yup [00:35:36] CristianCantoro: but the real beauty of jekyll is that it build static sites, so you technically wouldn't need it installed on a tool labs host. You could just build the static site locally and then upload the output to your tool's ~/public_html [00:38:46] PROBLEM - Host tools-webgrid-lighttpd-1407 is DOWN: CRITICAL - Host Unreachable (10.68.17.251) [00:40:58] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/राजा उप्रेती was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=851629 edit summary: [00:49:53] hey madhuvishy [00:50:00] my irc died [00:50:02] Krenair: yup [00:50:12] and my ssh client zoned out [00:50:17] the query didn [00:50:21] did not finish [00:50:25] rerunning [00:50:35] but its taking a really long time [00:52:23] the query is just supposed to find the time of the most recent occurrence of the issue [00:52:28] yeah [00:55:06] so I know whether we're just being affected by historical nonsense and need to do a one-time clean-up [00:55:12] or if this is an ongoing issue [01:08:53] madhuvishy: I think there's a 'deleted' flag [01:09:08] oh, except I guess you want deleted instances in this case? [01:09:43] ok, sorry, I'm not helping, I didn't read the backscroll thoroughly enough [01:11:42] This is one of those problems that won't happen when I'm looking for it. When I was at the last OpenStack conference the designate-sink devs said something about how their notification subscription (which should detect creation and deletion) turned out to be a dead end and they've totally replaced it now. [01:11:52] So I have my hopes somewhat pinned on the idea that that's what's biting us too [01:12:12] But I haven't dug in enough to know e.g. what the new solution is or when it was released. [01:13:52] * andrewbogott -> AFK again [01:16:05] Krenair: here you go [01:16:09] https://www.irccloud.com/pastebin/raLWrInq/ [01:16:47] hm [01:16:55] last one almost two weeks ago [01:18:39] yeah doesn't seem too historical [01:20:46] was hoping it'd be either several months ago or within the last 24 hours :/ [01:21:29] ha ha [01:31:08] madhuvishy, how far back to the designate logs on labservices1001/1002 go? [01:31:12] do the* [01:31:26] i have never been on that box - looking [01:37:35] Krenair: 2016-08-30 on labservices1001 [01:38:21] 2016-08-02 on 1002 [01:41:10] perhaps the logs from around the max(deleted_at) time could be found? [01:43:52] like `zgrep "2016-09-08 22:46:" /var/log/designate/designate-sink*.gz` [01:49:39] Krenair: haven't found anything yet [01:50:17] damn [01:50:27] Krenair: some api logs - but from 17:46:31 [01:50:41] so a completely different time of day? not much use [01:50:53] ya [01:51:26] when I ran the equivalent in labtest it showed me that labtest's last dns record issue like this happened after a different plugin of ours (the one that updates the ldap host entries) couldn't connect to ldap [01:56:40] i don't know what this is but designate-api logs for all 2016-09-08 17:46:31 looks like this [01:56:50] https://www.irccloud.com/pastebin/7puSfmZk/ [01:57:58] we're mainly looking at designate-sink [01:59:20] yeah that has nothing [02:02:19] I'm out of ideas [02:07:34] Krenair: yeah - let's call it a day - and look tomorrow [03:23:16] Hi. [03:23:32] Is https://dumps.wikimedia.org/enwiki/20160901/enwiki-20160901-pages-meta-current.xml.bz2 on tools-bastion-03? [03:23:45] We had /data/scratch/dumps at some point. [03:23:49] It seems to have gone missing, I guess. [03:24:52] I re-created /data/scratch/dumps/enwiki/ and I'm putting a dump in there. [03:25:17] Debra: why are you putting dumps in there instead of using the dumps from /public/dumps? [03:25:18] I can put it somewhere else if it's really a problem, but it looks like scratch has plenty of free space. [03:25:32] Oh, is there a /public/? [03:25:39] yup, has all the dumps [03:25:58] Ah, perfect, thanks! [03:26:21] yw [03:28:53] 10Striker, 06Community-Tech-Tool-Labs, 15User-bd808: Create Wikitech/LDAP accounts via a new user friendly guided workflow - https://phabricator.wikimedia.org/T144710#2654873 (10bd808) [03:31:49] I'm glad I asked in here. [03:32:35] I made some redirects: https://wikitech.wikimedia.org/wiki/Database_dumps && https://wikitech.wikimedia.org/wiki//public/dumps [03:36:14] /shared/mediawiki is also neat. I've been wondering about that as well. [03:36:44] thank you very much, Debra [03:36:54] Debra: use '/data/project/shared' instead of /shared. the latter is deprecated [03:36:59] (they point to the same thing) [03:40:08] Fair enough. I made the redirects point to the same section. [07:38:12] 06Labs, 10Labs-Infrastructure, 10DBA, 07Upstream: db1069: convert user_groups table to InnoDB across all the wikis - https://phabricator.wikimedia.org/T146121#2655239 (10Marostegui) ``` root@db1069:/srv# find . -name user_groups.frm | awk -F "." '{print $3}' | awk -F "/" '{print $1}' | uniq -c | sort -k2... [08:09:32] PROBLEM - Puppet staleness on tools-checker-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [43200.0] [09:20:58] 06Labs, 06Operations: Good bug reports - https://phabricator.wikimedia.org/T146266#2655420 (10Jishnugopim) [12:41:57] PROBLEM - SSH on tools-webgrid-lighttpd-1210 is CRITICAL: Server answer [13:04:32] Hi, I am trying to install jekyll on my toollabs project (wscontest) [13:04:39] but I get this error: [13:04:39] ERROR: Error installing jekyll: [13:04:40] jekyll requires Ruby version >= 2.0.0. [13:05:29] I tried to install rvm (https://rvm.io) but it needs some dependencies, it would be very convenient to have it [13:39:50] 06Labs, 10Tool-Labs: Add 3 webgrid-lighttpd trusty nodes to tools project - https://phabricator.wikimedia.org/T146212#2655885 (10chasemp) p:05Triage>03Normal [13:40:51] 06Labs, 10Tool-Labs: Add 3 webgrid-lighttpd trusty nodes to tools project - https://phabricator.wikimedia.org/T146212#2653821 (10chasemp) thanks @yuvipanda @madhuvishy you'll probably have to sync w/ @krenair or @Andrew on some DNS leak cleanup here :) [13:41:52] !log tools.heritage Reverting local changes to categorize_images.py (T146278). The CI mechanisms are there for a reason [13:41:53] T146278: categorize_images.py crashing due to malformated log message - https://phabricator.wikimedia.org/T146278 [13:41:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [13:45:59] 06Labs, 10Tool-Labs: Add 3 webgrid-lighttpd trusty nodes to tools project - https://phabricator.wikimedia.org/T146212#2655916 (10AlexMonk-WMF) We looked into it last night, but weren't able to find the cause. We do know that the last instance to leave a reverse DNS entry behind was deleted around 2016-09-08 22... [13:58:43] 06Labs, 10Labs-Infrastructure, 06Operations, 07Wikimedia-Incident: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2655948 (10AlexMonk-WMF) >>! In T146212#2655916, @AlexMonk-WMF wrote: > We looked into it last night, but weren't able to find the cause. We... [14:21:00] (03CR) 10Jean-Frédéric: [C: 04-1] "I have been able to test this locally (will update the ReadMe to outline how)." (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/310889 (https://phabricator.wikimedia.org/T144772) (owner: 10Lokal Profil) [15:57:16] (03PS2) 10Lokal Profil: Add Georgia in Georgian to database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/310889 (https://phabricator.wikimedia.org/T144772) [15:57:24] (03CR) 10Lokal Profil: Add Georgia in Georgian to database (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/310889 (https://phabricator.wikimedia.org/T144772) (owner: 10Lokal Profil) [15:58:48] (03CR) 10Jean-Frédéric: [C: 032] Add Georgia in Georgian to database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/310889 (https://phabricator.wikimedia.org/T144772) (owner: 10Lokal Profil) [16:00:01] (03Merged) 10jenkins-bot: Add Georgia in Georgian to database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/310889 (https://phabricator.wikimedia.org/T144772) (owner: 10Lokal Profil) [16:02:00] 06Labs, 10Tool-Labs: Add 3 webgrid-lighttpd trusty nodes to tools project - https://phabricator.wikimedia.org/T146212#2656327 (10madhuvishy) @AlexMonk-WMF With the offsite coming up and needing to add exec nodes to handle load, do you think we can do a clean up now so we can pool these, and then look into the... [16:16:14] madhuvishy, chasemp: I have a script that should clean this all up [16:16:20] I could run it if it's okay with you [16:16:57] Krenair: sure that would be great [16:16:59] Krenair: sounds good but let's wait for andrewbogott to be around just in case? [16:17:14] things go wrong that is [16:19:57] that sounds wise [16:20:08] i'll be back in a few, walking to office [16:20:22] madhuvishy, chasemp: it'll deal with existing entries, but won't prevent the same issue from recurring in future [16:21:05] I'd like to add some extra logging to sink_nova_fixed_multi [16:21:06] understood, def onboard for figuring this out but atm it's worth the bandaid I think [16:21:38] (at a level that makes it actually log stuff, unless andrew can figure out how to fix that) [16:25:54] (03CR) 10Lokal Profil: "> (1 comment)" (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/310889 (https://phabricator.wikimedia.org/T144772) (owner: 10Lokal Profil) [16:28:35] CristianCantoro: Your best bet for newer ruby support would be for us to get a ruby image build for the kubernetes webservices. We run debian jessie base images on kuberntes and that would provide Ruby 2.1.5 [16:29:47] (03CR) 10Lokal Profil: "> > (1 comment)" (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/310889 (https://phabricator.wikimedia.org/T144772) (owner: 10Lokal Profil) [16:36:06] Krenair: I'm here… you were going to clean up leaked dns entries? [16:36:51] yes [16:38:38] that's a script you've run before, right? Or is it something new? [16:39:26] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Add a easy way to run a ruby webservice on tools - https://phabricator.wikimedia.org/T141388#2496577 (10bd808) @MusikAnimal Debian Jessie has Ruby 2.1.5 packaged. Would that be new enough to replace your use of rbenv? What Ruby version are you using? I think Ruby coul... [16:40:14] andrewbogott, something new [16:40:22] well, ish [16:40:43] I wrote it over a month ago [16:41:03] ok — I'm happy for you to run it if you're confident. [16:41:07] always ran it with the write calls commented out, so it just said what it *would* do [16:41:26] ah, sure, if you've already done a dry run so you know what it'll do, that seems safe [16:46:59] andrewbogott, chasemp, madhuvishy: done [16:48:58] output: https://phabricator.wikimedia.org/P4089 [16:50:49] that's a lot of cleanup [16:51:01] we should ink this paste to the main task for historical ref? [16:51:27] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Add a easy way to run a ruby webservice on tools - https://phabricator.wikimedia.org/T141388#2656489 (10MusikAnimal) @bd808 Yeah that should suffice. I'm OK with something other than Unicorn too, especially if it means it will take less than 5 min... [16:52:56] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Clean up leaked designate entries - https://phabricator.wikimedia.org/T120797#2656505 (10AlexMonk-WMF) {P4089} [16:53:15] RECOVERY - Host tools-webgrid-lighttpd-1414 is UP: PING OK - Packet loss = 0%, RTA = 0.46 ms [16:53:16] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Clean up leaked designate entries - https://phabricator.wikimedia.org/T120797#2656510 (10AlexMonk-WMF) 05Open>03Resolved a:03AlexMonk-WMF [16:53:37] !log fastcci enable backports on fastcci-puppetmaster [16:53:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Fastcci/SAL, Master [16:53:54] yuvipanda: what did you figure out w/ backports? is it random where it's enable or a historical relic? [16:54:16] RECOVERY - Host tools-webgrid-lighttpd-1407 is UP: PING OK - Packet loss = 0%, RTA = 0.71 ms [16:54:22] chasemp: historical relic. we enabled it at some point, so older instances don't have it on [16:54:27] (enabled in the image) [16:54:39] thanks (was mainly curious) [16:54:59] !log monitoring enable backports on filippo-test-trusty [16:55:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Monitoring/SAL, Master [16:55:16] !log graphite enable backports on graphite-labs [16:55:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Graphite/SAL, Master [16:55:32] 06Labs, 10Labs-Infrastructure, 06Operations, 07Wikimedia-Incident: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2656525 (10AlexMonk-WMF) The script was run against real-labs in T120797 and most existing problem cases should be gone now [16:56:12] !log etcd enable backports on master [16:56:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Etcd/SAL, Master [16:56:57] RECOVERY - SSH on tools-webgrid-lighttpd-1210 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [16:57:08] !log mediawiki-core-team enabled backports on shaved-yak [16:57:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mediawiki-core-team/SAL, Master [16:57:31] !log tools reboot tools-webgrid-lighttpd-1407, tools-webgrid-lighttpd-1210, tools-webgrid-lighttpd-1414, and then tools-webgrid-lighttpd-1405 as the first 3 return [16:57:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [16:57:57] !log wikidata-query enable backports on wqds-puppetmaster [16:58:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-query/SAL, Master [16:58:12] 06Labs, 10Tool-Labs: Add 3 webgrid-lighttpd trusty nodes to tools project - https://phabricator.wikimedia.org/T146212#2656533 (10AlexMonk-WMF) Cleanup script has been run for existing cases. ```krenair@bastion-01:~$ host tools-webgrid-lighttpd-1418 tools-webgrid-lighttpd-1418.eqiad.wmflabs has address 10.68.2... [16:58:52] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Clean up leaked designate entries - https://phabricator.wikimedia.org/T120797#1861262 (10hashar) \O/ [16:59:13] RECOVERY - Puppet run on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [16:59:30] RECOVERY - Puppet run on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:40] My new labs instance is having problems with puppet: 2016-09-21T16:54:28.235782+00:00 captcha-ai-02 rc.local[406]: #033[1;31mError: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to determined $::labsproject at /etc/puppet/manifests/realm.pp:58 on node captcha-ai-02.privpol-captcha.eqiad.wmflabs#033[0m [17:01:05] And this: 2016-09-21T16:56:02.871263+00:00 captcha-ai-02 puppet-agent[519]: Could not request certificate: getaddrinfo: Name or service not known [17:01:24] tom29739: is this a project w/ a intra-project master? [17:01:30] Nope. [17:01:35] It uses the labs puppetmaster. [17:01:50] well that's super interesting [17:02:26] I recall a couple of my other instances having that same problem ages ago. [17:02:34] andrewbogott: ^ any reason project lookup would be failing here? seems like the race condition we know about but in a new place [17:02:43] But those instances had reused names, whereas this one doesn't [17:03:04] tom29739: is it ssh-able? [17:03:42] It asks for a password. [17:04:20] I'll look [17:08:01] RECOVERY - Puppet staleness on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [3600.0] [17:08:21] RECOVERY - Puppet run on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [17:08:46] Interestingly, it doesn't seem to have sent any Graphite data: https://tools.wmflabs.org/nagf/?project=privpol-captcha#h_captcha-ai-02_cpu [17:09:39] RECOVERY - Puppet staleness on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [3600.0] [17:10:27] RECOVERY - Puppet staleness on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [3600.0] [17:15:07] tom29739: if it's asking for a password it probably broke in the middle of the first puppet run and is generally busted [17:15:14] chasemp, tom29739, this looks like the same hostname failure as usual… tom29739, do you mind just rebuilding? Or is this happening consistently? [17:16:41] andrewbogott: my main thought was extra bonus points for not happening in a project w/ its own master [17:16:52] we'll have to dig into this post offiste man, this is like 6 times in a week or so [17:16:56] (that I know of) [17:26:19] !log tools.admin Removed Mark Bergsma, Ryan Lane, and Coren from maintainers list. If any of them want back on I'll be glad to re-add them. [17:26:23] mark: ^ [17:26:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL, Master [17:31:11] andrewbogott, the new new instance seems to be working. Thanks :) [18:42:45] !log tools Repooled tools-webgrid-lighttpd-1416 (T146212) after dns records cleanup [18:42:47] T146212: Add 3 webgrid-lighttpd trusty nodes to tools project - https://phabricator.wikimedia.org/T146212 [18:42:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [18:43:21] madhuvishy: can you run a few test on those new nodes to validate them? :) [18:43:43] chasemp: yeah - the repool script ran a test - it went okay [18:43:46] sweet [18:43:47] i'm watching it to see [18:43:59] I don't know that the test thing would have caught the errors before? [18:44:02] main reason I asked [18:44:09] it was work type specific I think [18:44:17] yeah - i think the issues happened when webservice start [18:45:57] but it should be okay - there are no duplicate records in designate [18:47:14] tools.jembot is scheduled there [18:56:42] !log tools Repooled tools-webgrid-lighttpd-1418 (T146212) after dns records cleanup [18:56:43] T146212: Add 3 webgrid-lighttpd trusty nodes to tools project - https://phabricator.wikimedia.org/T146212 [18:56:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [18:57:52] 10Labs-Kubernetes, 06Community-Tech, 06Wikisource: Make Google OCR API on Tool Labs work under Kubernetes - https://phabricator.wikimedia.org/T146311#2657046 (10kaldari) [19:37:20] PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:03:54] 06Labs, 10Tool-Labs: Add 3 webgrid-lighttpd trusty nodes to tools project - https://phabricator.wikimedia.org/T146212#2657258 (10madhuvishy) 05Open>03Resolved This is all done - tools-webgrid-lighttpd-1415, 1416, 1418 are up and running. [20:27:21] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.93 ms [20:29:57] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [20:32:10] 06Labs, 07Tracking: Increase quota for tools project - https://phabricator.wikimedia.org/T146322#2657383 (10madhuvishy) [20:47:28] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.84 ms [20:49:05] 06Labs, 10Labs-Infrastructure: Intermittent ldap failures in designate-sink - https://phabricator.wikimedia.org/T146325#2657460 (10Andrew) [20:51:04] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [20:51:22] 06Labs, 10Labs-Infrastructure: Intermittent ldap failures in designate-sink - https://phabricator.wikimedia.org/T146325#2657487 (10Andrew) It looks to me like my code is racing to complete a transaction before the connection is closed on the server end. I know that we have something automatically cleaning up... [20:53:10] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.72 ms [20:56:08] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [21:26:26] 06Labs, 10Labs-Infrastructure: Intermittent ldap failures in designate-sink - https://phabricator.wikimedia.org/T146325#2657596 (10Andrew) OK, now I think that these designate_sink plugins are being run in a multi-threaded environment. I'm going to rewrite this one to be thread safe and see if that makes the... [21:38:09] 10Labs-Kubernetes, 06Community-Tech, 06Wikisource: Make Google OCR API on Tool Labs work under Kubernetes - https://phabricator.wikimedia.org/T146311#2657640 (10DannyH) p:05Triage>03Normal [21:41:05] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, 06WMF-Legal: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#2657643 (10bd808) [21:47:30] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, and 4 others: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#2657656 (10bd808) a:03bd808 I have drafted two straw dog proposals and am proposing that they be discussed and refined... [22:13:08] * bd808 spams everyone with https://meta.wikimedia.org/wiki/Requests_for_comment/Abandoned_Labs_tools links [22:15:33] bd808: I've already read through it :P [22:17:00] tom29739: one part I would love to have some help figuring out: how and where do we call a consensus vote for tool labs? [22:17:41] That's a good question [22:17:48] I did a vote on phab previously, but legoktm and others said they didn't like that [22:18:06] It's clear that the RFC didn't work very well [22:18:25] Maybe just something simple [22:18:26] yeah. it didn't get much participation [22:18:41] Central notice maybe? [22:19:01] Put it in the tool labs SSH notice? [22:19:09] One thing that seems to work on wikitech-l is posting that something is a done deal and then waiting a week for people to freak out :) [22:20:44] I think one of the problems with reaching the tool maintainers is that they really are on their home wikis and not irc/wikitech/mailing lists [22:21:32] there are certainly a few folks who are active here on irc but numerically not a very high percentage [22:21:48] and meta is ... not my favorite place to hang out [22:23:52] Meta isn't the best place [22:25:17] bd808, how have consensus votes been done in the past? [22:25:52] I'm honestly not sure its ever really been done. The only one I know of was the slowvote I did on Phab [22:26:23] That was https://phabricator.wikimedia.org/V7 [22:30:00] tom29739: I may just survey some of the existing on-wiki voting systems and arbitrarily propose one for wikitech. We need a community :/ [22:31:17] That's the trouble I find with Labs as a whole. [22:31:27] The community doesn't live on wikitech. [22:31:51] It lives elsewhere, like on the maintainers home wikis [22:32:53] Which presents a problem in some cases, because if the maintainer is German say, and puts his documentation on the German wiki.. [22:32:54] I can't speak German. [22:33:20] Something like SecureVote would probably work [22:33:31] Just changing it to make it insecure [22:40:22] tom29739: votes aren't a good way to decide things. [22:41:29] I kinda disagree with voting myself [22:42:11] This is good: Editors might miss the best solution (or the best compromise) because it wasn't one of the options. This is especially problematic when there are complex or multiple issues involved. Establishing consensus requires expressing that opinion in terms other than a choice between discrete options, and expanding the reasoning behind it, addressing [22:42:11] the points that others have left, until all come to a mutually agreeable solution. [22:45:04] a vote is not the proper way for something so open as this [22:45:08] 06Labs, 10Labs-Infrastructure: Intermittent ldap failures in designate-sink - https://phabricator.wikimedia.org/T146325#2657837 (10Andrew) https://gerrit.wikimedia.org/r/#/c/312127/ [22:56:22] I think mostly what I'm looking for a small number of "omg this is the worst" responses honestly [22:56:57] like if 85% of people who actually do respond don't hate the whole thing then it's probably fine [22:57:33] But I really have never figured out how consensus is gauged [23:12:27] 06Labs, 10Labs-Infrastructure: Intermittent ldap failures in designate-sink - https://phabricator.wikimedia.org/T146325#2657888 (10Andrew) 05Open>03Resolved Seems happy now! [23:43:26] 06Labs: certcleaner.py uses ldap instance records. Should instead talk to keystone and nova - https://phabricator.wikimedia.org/T146303#2657921 (10AlexMonk-WMF) This isn't run by labs instances, is it? Just the main puppetmaster on labcontrol?