[00:38:15] <wikibugs>	 06Labs, 06Discovery, 06Maps: Update, replace or delete maps-tiles1 - https://phabricator.wikimedia.org/T157708#3013957 (10scfc) `tiles.wmflabs.org` is served by the instance `maps-tiles3` in the project `maps`:  ``` sqlite> SELECT project.name, url FROM backend JOIN route ON route_id = route.id JOIN project...
[00:52:18] <wikibugs>	 06Labs, 10Tool-Labs: replica.my.cnf and .kube/config is not readable by the group members - https://phabricator.wikimedia.org/T158581#3041806 (10scfc) 05Open>03declined I think it can be surprising as I believe it is intended that maintainer accounts have complete and transparent control over tool account'...
[01:42:32] <wikibugs>	 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3041865 (10Reedy)
[01:43:56] <wikibugs>	 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3041878 (10Reedy)
[01:44:28] <wikibugs>	 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3041865 (10Reedy)
[01:47:44] <wikibugs>	 06Labs, 10wikitech.wikimedia.org: Cleanup $wgObjectCaches for wikitech - https://phabricator.wikimedia.org/T158614#3041883 (10Reedy)
[01:47:56] <wikibugs>	 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3041897 (10Reedy)
[01:47:58] <wikibugs>	 06Labs, 10wikitech.wikimedia.org: Cleanup $wgObjectCaches for wikitech - https://phabricator.wikimedia.org/T158614#3041896 (10Reedy)
[01:49:00] <wikibugs>	 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3041865 (10Reedy) ``` reedy@silver:~$ mwscript eval.php labswiki > echo $wgMainCacheType; memcached-pecl > var_dump( $wgObjectCac...
[01:50:16] <wikibugs>	 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3041899 (10Reedy)
[02:04:10] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1413 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[02:39:12] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0]
[03:40:50] <halfak>	 !log ores restarted uwsgi and precached on ores-web-03
[03:40:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL
[03:59:57] <wikibugs>	 (03PS1) 10Krinkle: frontend: Un-hide submit button once form fields change [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/338922
[04:00:19] <wikibugs>	 (03CR) 10Krinkle: [C: 032] frontend: Un-hide submit button once form fields change [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/338922 (owner: 10Krinkle)
[04:01:12] <wikibugs>	 (03Merged) 10jenkins-bot: frontend: Un-hide submit button once form fields change [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/338922 (owner: 10Krinkle)
[04:06:13] <wikibugs>	 (03PS1) 10Krinkle: frontend: Fixup 0ce2136 [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/338923
[04:15:52] <wikibugs>	 (03CR) 10Krinkle: [C: 032] frontend: Fixup 0ce2136 [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/338923 (owner: 10Krinkle)
[04:19:39] <wikibugs>	 (03PS1) 10Krinkle: frontend: Move <script scc> request earlier (but with 'defer') [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/338924
[04:21:10] <wikibugs>	 (03PS1) 10Krinkle: frontend: Fixup 0ce2136 [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/338925
[04:21:16] <wikibugs>	 (03CR) 10Krinkle: [C: 032] frontend: Move <script scc> request earlier (but with 'defer') [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/338924 (owner: 10Krinkle)
[04:21:19] <wikibugs>	 (03CR) 10Krinkle: [C: 032] frontend: Fixup 0ce2136 [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/338925 (owner: 10Krinkle)
[04:21:43] <wikibugs>	 (03Merged) 10jenkins-bot: frontend: Move <script scc> request earlier (but with 'defer') [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/338924 (owner: 10Krinkle)
[04:22:37] <wikibugs>	 (03Merged) 10jenkins-bot: frontend: Fixup 0ce2136 [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/338925 (owner: 10Krinkle)
[05:05:11] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1413 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[05:05:52] <Tr8dr>	 What's the easiest way to expose files in my Tool Labs home directory on a static http(s) URL?
[05:40:10] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0]
[06:38:43] <wikibugs>	 (03PS1) 10BryanDavis: Use IB3 library for basic functionality [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/338933
[06:42:37] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1406 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[06:57:39] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1408 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[07:22:39] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0]
[07:37:39] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0]
[07:55:14] <wikibugs>	 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Pywikibot bot" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134495#2267300 (10Xephyr826) a:03Xephyr826 I'd like to this give this one a try.
[11:13:25] <wikibugs>	 06Labs, 10Tool-Labs-tools-Other: `fr-wikiversity` Tool should get deleted - https://phabricator.wikimedia.org/T133778#2242872 (10Youni) Sorry, i'm not so easy with phabricator. I confirm that  '' fr-wikiversity''  is to be deleted.  Sorry for the mistake, i'm now working on '' fr-wikiversity-ns''. Thanks.
[13:40:48] <wikibugs>	 10Tool-Labs-tools-Other: Migrate https//toolserver.org/~wiegels/wikipedia-termine.php to Tool Labs - https://phabricator.wikimedia.org/T62888#3042726 (10Ricordisamoa)
[13:42:42] <wikibugs>	 06Labs: Request creation of wikidiff2-wmde-dev labs project - https://phabricator.wikimedia.org/T158645#3042748 (10jkroll)
[14:54:07] <wikibugs>	 06Labs, 10Labs-Infrastructure: Audit use of 'novaadmin' within keystone and ldap - https://phabricator.wikimedia.org/T158650#3042929 (10Andrew)
[14:55:17] <wikibugs>	 06Labs, 10Labs-Infrastructure: Audit use of 'novaadmin' within keystone and ldap - https://phabricator.wikimedia.org/T158650#3042929 (10chasemp) k8s-master-01:~# getent group project-chasetestproject project-chasetestproject:*:53308:yuvipanda,rush,novaadmin
[14:55:25] <wikibugs>	 06Labs, 10Labs-Infrastructure: Audit use of 'novaadmin' within keystone and ldap - https://phabricator.wikimedia.org/T158650#3042958 (10chasemp) p:05Triage>03Normal
[15:20:55] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[15:39:37] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#3043185 (10Andrew)
[16:00:57] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:36:15] <wikibugs>	 (03CR) 10BryanDavis: [C: 032] Guard against multiple bots competing in channels [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/338048 (https://phabricator.wikimedia.org/T156895) (owner: 10BryanDavis)
[16:41:38] <wikibugs>	 (03Merged) 10jenkins-bot: Guard against multiple bots competing in channels [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/338048 (https://phabricator.wikimedia.org/T156895) (owner: 10BryanDavis)
[16:42:16] <wikibugs>	 (03CR) 10BryanDavis: [C: 032] Use IB3 library for basic functionality [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/338933 (owner: 10BryanDavis)
[16:44:08] <wikibugs>	 (03Merged) 10jenkins-bot: Use IB3 library for basic functionality [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/338933 (owner: 10BryanDavis)
[16:44:44] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#3043382 (10hashar) >>! In T143349#3027754, @demon wrote: >>>! In T143349#3026704, @MaxSem wrote: >> I'm strongly in favor of #1. At this point, 1.23 is likely only to have security updat...
[16:45:07] <bd808>	 !log tools.stashbot Updating to 39ca416 Use IB3 library for basic functionality
[16:45:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[16:48:44] <bd808>	 !log tools.stashbot Restarted
[16:48:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[17:00:04] <wikibugs>	 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3043530 (10JAllemandou) @Marostegui : Quick update on that one as well - How many shards are still to be done? Thanks !
[17:06:41] <wikibugs>	 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3043562 (10Marostegui) >>! In T153743#3043530, @JAllemandou wrote: > @Marostegui : Quick update on that one as well - How many shards are still to be done?...
[17:20:02] <wikibugs>	 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3043611 (10JAllemandou) Thanks for the headsup @Marostegui  :)
[17:38:49] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/For Developers was modified, changed by Federhalter link https://www.mediawiki.org/w/index.php?diff=2404853 edit summary: /* Demo Time! */ Fixed typo in comment
[18:28:44] <wikibugs>	 10Tool-Labs-tools-Pageviews: Massviews export data is incorrect (both CSV and JSON) - https://phabricator.wikimedia.org/T158505#3043936 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal Thanks for the report! This should now be fixed, and also for Langviews, Redirect Views and Userviews. Let me know if you...
[18:34:03] <wikibugs>	 10Tool-Labs-tools-Pageviews: Massviews export data is incorrect (both CSV and JSON) - https://phabricator.wikimedia.org/T158505#3043948 (10MusikAnimal) I should clarify: Now for older dates, if the data is not available, a zero is filled in, which is safe assumption due to the way the API works. However any blan...
[18:36:55] <J-Mo>	 bd808, yuvipanda: getting a 502 bad gateway error on PAWS again this morning. anything you can do?
[18:37:02] <J-Mo>	 or, anything I can do? ;)
[18:41:39] <wikibugs>	 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries: nlwiki Labs replica with problems - https://phabricator.wikimedia.org/T138927#3043967 (10Sjoerddebruin) 05Resolved>03Open It shouldn't be that this page and two **privacy violations** keep appearing on pages as https://tools.wmflabs.org/wikidata-to...
[18:49:14] <chasemp>	 J-Mo1: I haven't seen yuvi around yet today but I'll make a task quick as I'm not sure what's up
[18:49:30] <J-Mo1>	 thanks chasemp
[18:50:04] <wikibugs>	 06Labs, 10PAWS: paws returns 502 bad gateway - https://phabricator.wikimedia.org/T158685#3043978 (10chasemp)
[18:50:25] <wikibugs>	 06Labs, 10PAWS: paws returns 502 bad gateway - https://phabricator.wikimedia.org/T158685#3043978 (10chasemp) p:05Triage>03Normal
[19:36:53] <madhuvishy>	 J-Mo1: check now?
[19:37:44] <J-Mo1>	 madhuvishy it works! thanks
[19:38:41] <madhuvishy>	 cool :) np
[19:43:45] <wikibugs>	 06Labs, 10PAWS: paws returns 502 bad gateway - https://phabricator.wikimedia.org/T158685#3043978 (10madhuvishy) Looked into hub logs - was seeing a stream of these errors. Not sure why this is happening, and why it takes the hub down - but restarting the hub seemed to restore normalcy, at-least for now.   ```...
[19:57:49] <wikibugs>	 06Labs, 10PAWS: paws returns 502 bad gateway - https://phabricator.wikimedia.org/T158685#3043978 (10valhallasw) I had a hunch there might be two distinct users with different capitalization, but that doesn't seem to be the case. There /are/ a whole bunch of these usernames: Info-farmer, Info-farmer-bio-bot, In...
[21:08:41] <wikibugs>	 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3018288 (10Mattflaschen-WMF) Is this the reason for "The requested URL /browser/index.php was not found on this server." at http://wm-bot.wmflabs.org/browser/index.php?display=%23wikimedia-collaboration , or should I f...
[21:11:23] <wikibugs>	 10Wikibugs: Wikibugs always points to the creation of the task - https://phabricator.wikimedia.org/T158704#3044585 (10Luke081515)
[22:42:19] <wikibugs_>	 (03Draft2) 10Zppix: Add notifications for tools.zppixbot and quarrybot-enwiki [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/339074
[22:47:26] <wikibugs_>	 (03CR) 10jerkins-bot: [V: 04-1] Add notifications for tools.zppixbot and quarrybot-enwiki [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/339074 (owner: 10Zppix)
[22:48:29] <wikibugs_>	 (03PS3) 10Zppix: Add notifications for tools.zppixbot and quarrybot-enwiki [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/339074
[22:54:03] <bd808>	 !log tools.jouncebot Cherry-picked {{gerrit|339066}} for testing
[22:54:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jouncebot/SAL
[22:54:27] <wikibugs_>	 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 05Prometheus-metrics-monitoring: Labs Prometheus not recording k8s stats since 2017-01-24T06:00 - https://phabricator.wikimedia.org/T157355#3044916 (10scfc) @bd808, with the dashboard now showing values, is this task resolved for you or do you want continuous data f...
[22:54:49] <Zppix>	 bd808: i'd go ahead and merge the config change i made when/if you deploy 339066
[22:55:16] <Zppix>	 Deploy*
[22:55:32] <Zppix>	 wow wheres brain cells when you need them
[23:09:50] <bd808>	 !log tools.jouncebot All is lost! Or at least things are borked after touching the virtualenv. Investigating.
[23:09:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jouncebot/SAL
[23:10:38] <Zppix>	 Uh oh
[23:10:46] <Zppix>	 bd808: well then
[23:15:31] <bd808>	 Zppix: no worries. I've got 43 more minutes to get it running :)
[23:15:55] * Zppix times bd808
[23:16:29] <wikibugs_>	 06Labs, 10Tool-Labs: Puppet fails on tools-docker-builder-03 - https://phabricator.wikimedia.org/T157415#3044988 (10scfc) I have deleted `tools-docker-builder-03`.
[23:16:48] <wikibugs_>	 10Tool-Labs-tools-Other, 10Deployment-Systems, 13Patch-For-Review, 15User-bd808: Jouncebot not joining #wikimedia-operations - https://phabricator.wikimedia.org/T144189#3044990 (10bd808) 05Open>03Resolved a:03bd808 Fixed a long time ago
[23:18:38] <wikibugs_>	 10Tool-Labs-tools-Other: Jouncebot failing to parse Deployments page - https://phabricator.wikimedia.org/T158715#3045001 (10bd808)
[23:18:44] <shinken-wm>	 PROBLEM - Host tools-docker-builder-03 is DOWN: CRITICAL - Host Unreachable (10.68.19.21)
[23:20:35] <chasemp>	 ^ yuvipanda when you are back, is this an issue?
[23:22:58] <Zppix>	 chasemp: https://phabricator.wikimedia.org/T157415#3044988
[23:37:08] <wikibugs_>	 06Labs, 13Patch-For-Review: track growth of project and home directories - https://phabricator.wikimedia.org/T126623#2018951 (10madhuvishy) Graphite now logs tools and misc NFS share directory sizes at labstore.labstore-secondary.DirectorySizeCollector on prod. Grafana dashboard based on this data - https://gr...
[23:38:21] <wikibugs_>	 06Labs, 10Tool-Labs: deb.tools.wmflabs.org is not accessible from outside Tool Labs - https://phabricator.wikimedia.org/T158383#3045054 (10scfc) I have switched `role::aptly::client::servername` in https://wikitech.wikimedia.org/wiki/Hiera:Tools to `tools-services-01.tools.eqiad.wmflabs` with no apparent probl...
[23:47:27] <wikibugs_>	 10Tool-Labs-tools-Other: Jouncebot failing to parse Deployments page - https://phabricator.wikimedia.org/T158715#3045074 (10bd808) Here's the real problem: ``` 2017-02-21 16:36:01,720 - ERROR - Could not fetch page due to exception: HTTPError(u'414 Client Error: Request-URI Too Long for url: https://wikitech.wik...