[00:13:40] 10Tool-Labs-tools-Pageviews: Add option to include redirect views in pageviews - https://phabricator.wikimedia.org/T163621#3216406 (10MusikAnimal) This was declined with T128495 but times have changed, and the API is much faster now. Even so, querying for redirects could slow things down dramatically depending o... [00:14:19] 10Tool-Labs-tools-Pageviews: Add option to include redirect views in pageviews - https://phabricator.wikimedia.org/T163621#3216414 (10MusikAnimal) [00:14:22] 10Tool-Labs-tools-Pageviews, 06Community-Tech: Add option to pageviews tool to show stats for all redirects to a page as well - https://phabricator.wikimedia.org/T128495#2076747 (10MusikAnimal) [00:22:42] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:27:41] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Optimize edit count queries in XTools - https://phabricator.wikimedia.org/T163284#3216431 (10kaldari) It looks like we don't use the `user_editcount` column because it is considered "approximate", which begs the question, why is it approximate? My edit counts... [00:50:48] PROBLEM - Puppet errors on tools-exec-1442 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:02:44] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [01:18:12] I have a problem with my Webservice [01:20:49] After trying to use HHVM with lighttpd and standalone, I'm unable to stop the Web service [01:25:49] RECOVERY - Puppet errors on tools-exec-1442 is OK: OK: Less than 1.00% above the threshold [0.0] [01:40:18] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Optimize edit count queries in XTools - https://phabricator.wikimedia.org/T163284#3192230 (10MusikAnimal) >>! In T163284#3216431, @kaldari wrote: > It looks like we don't use the `user_editcount` column because it is considered "approximate", which begs the qu... [02:02:28] I already resolved it [02:03:17] But I'm still finding a way to use hhvm in Web server (and I'm running the Webserver at kubernetes) [02:24:57] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [02:59:58] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [03:20:57] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [03:21:21] can anyone tell me why croptool (Commons) is not working? [03:33:47] i get disconnected from IRC everytime i click the tool :/ [03:55:58] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [04:42:11] 06Labs, 10Labs-Infrastructure: Audit disk usage on labvirts - https://phabricator.wikimedia.org/T163796#3216632 (10Andrew) For a quick start, I'm attaching a spreadsheet {F7786435} that shows instance allocated usage (from the instance flavor) next to actual instance disk space usage (via 'du' on the labvirts.... [05:46:41] WaRpAtH: it looks your tool is running but failing with errors, have you checked the error logs by logging into your tool account? [06:52:31] PROBLEM - Puppet errors on tools-exec-1424 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:27:32] RECOVERY - Puppet errors on tools-exec-1424 is OK: OK: Less than 1.00% above the threshold [0.0] [07:32:10] 06Labs, 10Horizon: Horizon loses credentials every day - https://phabricator.wikimedia.org/T145703#2638694 (10Nikerabbit) I logged in yesterday with "Remember me" and today I had to log-in again, so doesn't seem to be fixed. [07:43:24] 06Labs, 10Horizon: Horizon loses credentials every day - https://phabricator.wikimedia.org/T145703#2638694 (10jcrespo) Could it be related (from the list)? ``` The switch maintenance that we were bracing for didn't happen today. I've re-enabled normal functionality on Horizon and Wikitech, but will disable t... [08:55:38] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Create a cronjob/check to run check_private_data data script and report back - https://phabricator.wikimedia.org/T153680#3216979 (10Marostegui) 05Open>03Resolved a:03Marostegui Going to close this for now as the script is working fine as a fir... [09:24:44] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, and 2 others: Run a documentation sprint for Labs - https://phabricator.wikimedia.org/T101659#3217047 (10Qgil) Can any of these subtasks be connected with the WriteTheDocs activities coming to San Francisco and Portland? [09:42:23] 06Labs, 10Tool-Labs, 10DBA: labsdb1001 and labsdb1003 short on available space - https://phabricator.wikimedia.org/T132431#3217063 (10jcrespo) 05Open>03Resolved We added 1 extra terabyte by deleting /srvuserdata on both hosts- this will likely impact performance negatively, but at leasy they can now rece... [12:47:57] 06Labs, 10Labs-Infrastructure, 10Tool-Labs-tools-Other, 15User-bd808: Automatically updated list of all configured domains - https://phabricator.wikimedia.org/T45580#3217347 (10Nemo_bis) Thanks. That's quite useful. [13:38:28] 06Labs, 06Operations, 10wikitech.wikimedia.org: Update wikitech-static and develop procedures to keep it maintained - https://phabricator.wikimedia.org/T163721#3217467 (10Andrew) I updated the password so that's all good now. [13:59:17] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Shorter token life for novaobserver/novaadmin - https://phabricator.wikimedia.org/T163259#3217538 (10Andrew) [13:59:20] 06Labs, 10Labs-Infrastructure, 05MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), 13Patch-For-Review: Support project creation without OpenStackManager - https://phabricator.wikimedia.org/T150091#3217539 (10Andrew) [13:59:22] 06Labs, 10Labs-Infrastructure: Wikitech error when adding users to projects - https://phabricator.wikimedia.org/T159021#3217536 (10Andrew) 05Open>03Resolved a:03Andrew [14:03:23] 06Labs, 10Horizon: Stop requiring two-factor authentication for horizon.wikimedia.org - https://phabricator.wikimedia.org/T161473#3217567 (10Andrew) >>! In T161473#3204596, @Zppix wrote: > I dislike the 2FA requirement I do things from many different devices and I dont have access all the time to internet on m... [14:06:17] 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3217576 (10Andrew) The stacktrace doesn't implicate anything that's not running on all the other wikis... does anyone have a theo... [14:07:55] 06Labs, 10Tool-Labs: Freenode sometimes throttles bot connections from tools - https://phabricator.wikimedia.org/T151704#3217599 (10Andrew) Nope, I never heard anything back. [14:41:21] andrewbogott: I've got a quick question regarding the grid? [14:42:23] When I submit a job to the grid, and direct the output to the respective files, is it possible to send the regular out and err to the same file? [14:42:34] Or will that create IO issues? [14:49:42] Cyberpower678: I wouldn't expect it to cause an IO issues (other than interleaved output) [14:49:49] but I don't know what command you'd use [14:50:38] Well I would simply point -e to the same file as -o. I'm getting freaky errors on one of my jobs, and I can't replicate it anywhere. [14:51:04] But for some reason the script just completely goes insane after a bit while working on a job. [14:52:21] So I want to place them in the same file to get some chronology. I've tried pasting error messages and data points to echo out, but that didn't help. I've been spinning my wheels on this for days now. [15:06:00] 06Labs, 06Operations: Update documentation for Tools Proxy failover - https://phabricator.wikimedia.org/T163390#3217866 (10Andrew) Looks good to me. I added clarification about short-term failovers since that's the most frequent use case: https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource%3ATools... [15:06:26] 06Labs, 06Operations: Update documentation for Tools Proxy failover - https://phabricator.wikimedia.org/T163390#3217867 (10Andrew) a:05Andrew>03chasemp [15:11:09] 06Labs, 10Labs-Infrastructure: Audit disk usage on labvirts - https://phabricator.wikimedia.org/T163796#3217904 (10Andrew) labvirt1001.eqiad.wmnet: /dev/sdb1 2.2T 381G 1.9T 18% /var/lib/nova/instances labvirt1002.eqiad.wmnet: /dev/sdb1 2.2T 1.2T 1.1T 53% /var/lib/nova/instances labvir... [15:16:32] 06Labs, 10Labs-Infrastructure: Audit disk usage on labvirts - https://phabricator.wikimedia.org/T163796#3217927 (10Andrew) Worst case scenario: What if every one of those instances suddenly partitioned and filled every bit of allocated space? Labvirt1007 and 1008 would choke. Labvirt1003, 1004, 1005, 1006 w... [15:29:41] it does open teh file and write rather than say stderr and stdout redirect so it could be a race in there somewhere but it's not likely and I would risk it for debug certainly [15:41:42] 06Labs, 06Operations: During labservices1001 failover fqdn changed from foo.project.eqiad.wmflabs to foo.eqiad.wmflabs - https://phabricator.wikimedia.org/T163823#3218042 (10Andrew) When the metaservice is fully down then puppet errors out. If instead the metadata service is responding and returning an empty... [15:52:22] 10Tool-Labs-tools-Xtools, 10Addwiki, 03Community-Tech-Sprint: MediawikiApi::newFromPage() fails on non-XML HTML - https://phabricator.wikimedia.org/T163527#3218127 (10Addshore) >>! In T163527#3216021, @kaldari wrote: > Now we just need to wait for Addshore to cut a new version. 2.3.0 now released :) [15:59:27] 10Labs-project-Wikistats: wikistats: add new wikipedias: kbp, khw, dty and pt.wikimedia - https://phabricator.wikimedia.org/T160947#3218186 (10Dzahn) [16:15:25] 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3218242 (10bd808) >>! In T158613#3217576, @Andrew wrote: > The stacktrace doesn't implicate anything that's not running on all th... [16:17:07] 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3218262 (10Andrew) So the reason this doesn't make MediaWiki hhvm-only is because normal non-WMF deploys of mediawiki don't inclu... [16:17:54] 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3218263 (10Andrew) (obviously I vote for option 1, 'switch from PHP5 to HHVM for serving wikitech' since that one is forward and... [16:23:24] 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3218273 (10bd808) [16:48:16] 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3218329 (10bd808) >>! In T158613#3218262, @Andrew wrote: > So the reason this doesn't make MediaWiki hhvm-only is because normal... [16:50:10] 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3218338 (10bd808) [16:53:41] 06Labs, 10Labs-Infrastructure: Switch primary labservices node back to labservices1001 - https://phabricator.wikimedia.org/T164014#3218347 (10Andrew) [16:57:14] 10Tool-Labs-tools-Xtools, 10Addwiki, 03Community-Tech-Sprint: MediawikiApi::newFromPage() fails on non-XML HTML - https://phabricator.wikimedia.org/T163527#3218363 (10kaldari) @Addshore: Thanks!! [17:05:19] 06Labs, 06Operations, 13Patch-For-Review: Ensure we can survive a loss of labservices1001 - https://phabricator.wikimedia.org/T163402#3218381 (10Andrew) a:05Andrew>03None The particular maintenance that prompted this task is now complete. We still need to improve here, though. [17:32:41] how do I move https://gerrit.wikimedia.org/r/#/c/348201/ forward? [17:35:21] legoktm: iiuc the dba's already replicated it and that change looks good, we have asked for security to give their acknowledgment of making public new data via labsdb for awhile, can we ping those guys on teh task to thumbs up the creation of teh view? [17:35:39] and then we (ops) can do the rest I think [17:35:45] chasemp: security already approved it a long time ago, that's in the ticket [17:36:03] https://phabricator.wikimedia.org/T148583#2854927 [17:36:08] in https://phabricator.wikimedia.org/T160611 [17:36:09] ah [17:36:34] legoktm: can you link that ticket to the changset? [17:36:38] yes [17:37:18] I added it to the commit message [17:37:27] legoktm: cool then I'll run through this in teh afternoon today (or tomorrow morn at the latest) [17:37:31] thanks [17:37:58] the past few times it has taken some poking and proding so I want to leave a bit of time :) [17:38:49] 06Labs, 10DBA, 10MediaWiki-extensions-Linter, 13Patch-For-Review: Make "linter" table available on Labs - https://phabricator.wikimedia.org/T160611#3218500 (10chasemp) a:03chasemp Approved by security in https://phabricator.wikimedia.org/T148583#2854927 [17:50:43] bd808, BTW, your changes to mwoauth are released. docs will beupdated soon [17:51:27] 06Labs, 10DBA, 10MediaWiki-extensions-Linter, 13Patch-For-Review: Make "linter" table available on Labs - https://phabricator.wikimedia.org/T160611#3105094 (10Andrew) I merged the puppet change. Next we need to update things with ``` maintain-views --all-databases --table linter --debug --replace-all `... [17:54:09] halfak: nifty! [18:26:03] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Pages created interface gives error for most projects in new XTools - https://phabricator.wikimedia.org/T163634#3218758 (10MusikAnimal) >>! In T163634#3207437, @kaldari wrote: > @Samwilson: Is this a duplicate of T163527? Yep, this should be fixed once with a... [18:27:22] (03PS1) 10BryanDavis: Fix use of deprecated Logger.warn() method [labs/striker] - 10https://gerrit.wikimedia.org/r/350630 [18:56:32] (03PS1) 10BryanDavis: Switch logstash transport from UDP to TCP [labs/striker] - 10https://gerrit.wikimedia.org/r/350638 (https://phabricator.wikimedia.org/T151422) [19:02:24] 06Labs, 10Labs-Infrastructure: maintain-views needs to do something smarter when trying to update dbs that don't exist - https://phabricator.wikimedia.org/T164029#3218882 (10Andrew) [19:15:20] 06Labs, 10DBA, 10MediaWiki-extensions-Linter, 13Patch-For-Review: Make "linter" table available on Labs - https://phabricator.wikimedia.org/T160611#3218897 (10Andrew) This is now available on all db hosts except for 1001, which is misbehaving. [19:18:50] andrewbogott: remember that DNS error I emailed you about? [19:19:05] yes [19:19:18] It's literally killing my IABot workers. [19:19:20] https://phabricator.wikimedia.org/T163653 [19:19:39] Took me forever to localize the problem to the DNS servers on Labs. [19:19:42] andrewbogott: ^ [19:19:53] do you have a particular example of a thing that won't resolve, and where it won't resolve? [19:20:03] Because I would be getting 1,000,000 alerts if dns were actually down [19:21:01] No. It happens at random, and once it fails to resolve the first request, it fails to resolve ALL successive requests. [19:21:28] I actually have to restart the worker to get it to run again. [19:21:45] Otherwise it just stacks up all these errors. [19:21:56] Cyberpower678: we had some possible DNS disruption over the last 48h that should be solid now [19:22:03] is this a recent thing? [19:22:08] It's not. [19:22:11] It's still happening. [19:22:21] huh [19:23:02] I added Curl Error echoing on request failures. Because the problem doesn't go away until the script is restarted, I had to add a restarter to my code. [19:23:20] So when it detects this failure, it sleeps for 60 seconds and then restarts. [19:24:04] chasemp: it's also still happening on my Cyberbot project. [19:25:18] this seems like two problems atm for https://phabricator.wikimedia.org/T163653#3218896 [19:25:30] that's a bad failure mode and then lookup failure is obv not intended [19:26:28] chasemp: what do you mean? [19:27:30] failure could be more graceful in this case [19:27:55] and that isn't to say that the lookups should be failing so we'll have to get into that [19:28:35] When I designed it, I never anticipated DNS servers failing, so it wasn't something I targeted in trying to find out. [19:29:30] I also discovered, once the first curl request fails with this issue, no further curl requests would work. I would have tried to fix it some other way, but it seems only a script restarts does the trick. [19:29:48] do you know if DNS requests are timing out or returning nxdomain? [19:30:15] chasemp: nope. It just says can't resolve en.wikipedia.org [19:34:06] Cyberpower678: assuming you are not resolving other *.eqiad.wmflabs addresses you could try an external DNS for awhile? [19:34:10] to see [19:34:27] chasemp: how would I go about that? [19:35:08] see if using https://developers.google.com/speed/public-dns/ works out for a day or two [19:36:06] I know how to adjust the DNS servers on my own local devices. I use GDNS at home, as well as ODNS, but how do I do that on Labs? How do I set that up on tool labs? [19:37:48] chasemp: ^ [19:38:50] if these are instances you are administering you could change /etc/resolv.conf to debug or if in Tools I imgine using something like python-dns instead of the host resolvers is the only way [19:39:46] Okay, I'll change the file on my instances, but that still leaves toollabs, where the problem seems to be the worst. [19:40:23] chasemp: what is python-dns? [19:40:48] you need some way to query the resolvers directly if you want to test user different resolvers than the hsot [19:40:49] host [19:41:17] ... [19:41:44] I have never done that before. Are you saying I need my code to query the resolver directly? chasemp [19:42:24] are we sure that the failures are in error? It's not just that you're trying a lookup on a site that doesn't exist anymore? [19:43:47] API error encountered during page validation. Waiting 1 minute and restarting. [19:43:47] Curl Error: 6: Could not resolve host: en.wikipedia.org [19:43:49] andrewbogott: [19:43:55] ok [19:44:02] what host is this running on? [19:44:02] is that from Tools or your own project? [19:44:04] :) [19:44:08] Both [19:44:32] IABot is also sending error emails to IA API developers, who in turned forwarded me the message. [19:45:17] IABot usually sends error emails to the Wayback API devs so they can fix any issues. But the emails they have been getting also has the same resolving error. [19:46:19] Cyberpower678: you think this started monday? [19:46:48] * Cyberpower678 fetches an old email. [19:47:23] Cyberpower678: what host is this happening on? [19:47:27] Can you reproduce from the commandline? [19:47:55] andrewbogott: it's happening on tools for the most case. The CLI works correctly. When running on the grid, it fails quite randomly. [19:48:20] * chasemp meeting [19:48:24] chasemp: it's very possible. It wasn't as serious back then, but it's happening every 5 minutes now. [19:50:32] chasemp: oh wait. enwiki.labsdb, is being connected to. Changing the DNS resolver is going to break DB connections. [19:50:57] I will point out that sometimes it can't resolve to the DB either, but that's incredibly rare. [19:51:51] Cyberpower678: happen to know what node the job is scheduled on when this happens? [19:52:12] * Cyberpower678 looks. It's running and happening now, with the worker compensating. [19:52:29] andrewbogott: task@tools-exec-1407.eqiad.wmf [19:55:07] I've put a job on that host that does 'curl en.wikipedia.org' every second until it fails. We'll see [19:55:39] andrewbogott: one thought is we do some limited negative caching too via nscd I think [19:55:51] not for the root but maybe exacerbates [19:55:54] andrewbogott: I wish I could give you some triggers, but this seems to be happening totally at random and only when running on the grid. :/ [20:08:04] andrewbogott: /mnt/nfs/labstore-secondary-tools-project/iabot/Workers [20:08:13] You can peer into the output logs there. [20:09:05] Worker1 has the restarts in them. You can see that they just stop resolving, and the script restarts. [20:30:28] Cyberpower678: my test loop is still whirring happily [20:30:41] My script isn't. [20:33:05] andrewbogott: https://pastebin.com/vrv06JU4. This happens when the DNS fails on startup. [20:34:05] It's trying to connect to tools-db [20:35:02] Cyberpower678: I don't think I'll be able to help until you produce a trivial example that fails. Like e.g. a 3-line bash script. So far I just can't see any misbehavior that's not coupled to your code. [20:35:41] could it be the /rate/ of queries causing failures? how many queries a second are we talking? [20:35:55] (still in a meeting tho) [20:36:07] andrewbogott: I've looked through my code for the last 3 days to try and find a problem with it. The simple matter is, this problem doesn't happen on the labs' CLI, or on my local computer. [20:36:46] said 'trivial example' could of course be submitted to the grid if that's a necessary piece of the puzzle. [20:36:47] chasemp: I am uncertain how fast it is querying. It shouldn't be ridiculous. It's not doing anything in parallel. [20:37:02] andrewbogott: yes it is. [20:37:08] The CLI works fine. [20:49:42] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Erwin's-tools, 10Tool-Labs-tools-Other, 10DBA: s51362 has been rate limited to 2 concurrent connections for creating hundreds of 1400-second queries to labsdb1001 and labsdb1003 every 10 seconds - https://phabricator.wikimedia.org/T162519#3165938 (10Nemo_bis) Thanks... [20:51:20] Nemo_bis: thanks :-) [20:56:29] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Erwin's-tools, 10Tool-Labs-tools-Other, 10DBA: s51362 has been rate limited to 2 concurrent connections for creating hundreds of 1400-second queries to labsdb1001 and labsdb1003 every 10 seconds - https://phabricator.wikimedia.org/T162519#3219371 (10jcrespo) I have... [20:57:48] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Erwin's-tools, 10Tool-Labs-tools-Other, 10DBA: s51362 has been rate limited to 2 concurrent connections for creating hundreds of 1400-second queries to labsdb1001 and labsdb1003 every 10 seconds - https://phabricator.wikimedia.org/T162519#3219375 (10jcrespo) I would... [21:02:00] 06Labs, 06Operations, 10wikitech.wikimedia.org: Update wikitech-static and develop procedures to keep it maintained - https://phabricator.wikimedia.org/T163721#3219386 (10Andrew) [21:05:28] 06Labs, 06Operations, 10wikitech.wikimedia.org: Update wikitech-static and develop procedures to keep it maintained - https://phabricator.wikimedia.org/T163721#3219388 (10Andrew) I have three different sets of proposed answers: Option A: 1) Once a month 2) Someone in Ops (or, possibly, me) 3) Icinga should... [21:08:56] 10Tool-Labs-tools-Other: Unknown "cewbot" user lurking in channels - https://phabricator.wikimedia.org/T160907#3219394 (10Nemo_bis) 05duplicate>03Resolved p:05Triage>03Normal a:03Kanashimi [21:09:11] 10Tool-Labs-tools-Other: Unknown "cewbot" user lurking in channels - https://phabricator.wikimedia.org/T160907#3114242 (10Nemo_bis) Fixed by the addition of the description: > A bot running routine works on enwiki, zhwiki, jawiki, zhwikinews, and listen to #wikipedia-zh, #wikinews-zh, #mediawiki-i18n. Please con... [21:09:43] 10Tool-Labs-tools-Other: Unknown "cewbot" user lurking in channels - https://phabricator.wikimedia.org/T160907#3219403 (10Nemo_bis) [21:09:46] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: Shut down "cewbot" - https://phabricator.wikimedia.org/T160264#3219405 (10Nemo_bis) [21:29:14] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, and 2 others: Run a documentation sprint for Labs - https://phabricator.wikimedia.org/T101659#3219431 (10srishakatux) @Qgil Yes! This is happening. Planning on [[ https://www.mediawiki.org/wiki/Write_the_Docs_San_Francisco_Meetup_May_2... [21:29:57] 10Striker: Striker error when logging non-ASCII strings - https://phabricator.wikimedia.org/T164034#3219433 (10bd808) [21:56:50] 06Labs, 10DBA, 10MediaWiki-extensions-Linter, 13Patch-For-Review: Make "linter" table available on Labs - https://phabricator.wikimedia.org/T160611#3219478 (10chasemp) a:05chasemp>03Andrew thanks @Andrew [21:57:16] 06Labs, 06Operations: Update documentation for Tools Proxy failover - https://phabricator.wikimedia.org/T163390#3219480 (10chasemp) 05Open>03Resolved thanks, calling it good for now