[00:00:41] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [01:12:55] PROBLEM - Puppet errors on tools-exec-1435 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:52:56] RECOVERY - Puppet errors on tools-exec-1435 is OK: OK: Less than 1.00% above the threshold [0.0] [02:13:57] PROBLEM - Puppet errors on tools-exec-1435 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [02:23:26] PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [02:29:33] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [02:53:52] RECOVERY - Puppet errors on tools-exec-1435 is OK: OK: Less than 1.00% above the threshold [0.0] [02:58:27] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [03:09:31] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [04:40:25] 06Labs, 10DBA: Labs database corruption - https://phabricator.wikimedia.org/T166091#3285070 (10Legoktm) From a production database server: ``` mysql:wikiadmin@db1083 [enwiki]> SELECT pl_namespace, pl_title -> FROM page -> JOIN pagelinks ON pl_from = page_id -> WHERE page_namespace=0 AND page_title=... [06:03:13] 06Labs, 10DBA: Labs database corruption - https://phabricator.wikimedia.org/T166091#3285070 (10Marostegui) Looks like this is only happening on the old labs infra (, db1069 and labsdb1001 and labsdb1003). The new one are showing the same value as production. [06:31:03] PROBLEM - Puppet errors on tools-exec-1440 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:02:18] 06Labs: Access to *.labsdb servers needed for the wikidataconcepts project/instance - https://phabricator.wikimedia.org/T165926#3285441 (10Addshore) > we need access to MariaDB replicas for Labs project: wikidataconcepts, from instance: wikidataconcepts; requesting this in line with: https://wikitech.wikimedia.o... [07:11:05] RECOVERY - Puppet errors on tools-exec-1440 is OK: OK: Less than 1.00% above the threshold [0.0] [07:38:37] PROBLEM - Free space - all mounts on tools-proxy-02 is CRITICAL: CRITICAL: tools.tools-proxy-02.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-proxy-02.diskspace.root.byte_percentfree (<22.22%) [07:43:39] 06Labs: Access to *.labsdb servers needed for the wikidataconcepts project/instance - https://phabricator.wikimedia.org/T165926#3285510 (10GoranSMilovanovic) Thanks @Addshore already done (@bd808 provided the same suggestions on IRC). [08:50:42] 10Tool-Labs-tools-Other, 10DBA, 13Patch-For-Review: Tired of APIError: readonly - https://phabricator.wikimedia.org/T164191#3225024 (10Multichill) Yup, I'm tired of it too, but should be handled by Pywikibot. MediaWiki seems to have introduced a new way of throwing a readonly error and Pywikibot doesn't hand... [09:19:43] Hi, I would appreciate support on the following. I have a Labs instance (project: wikidataconcepts, instance: wikidataconcepts). I need to create a database somewhere on *.labsdb for an analytical application that I'm developing there. I couldn't find anything in the documentation that would help me figure out how to do it. Right now, I can only access replicas on *.labsdb. I am working as a Data Analyst for WMDE and the applic [09:23:37] RECOVERY - Free space - all mounts on tools-proxy-02 is OK: OK: tools.tools-proxy-02.diskspace._public_dumps.byte_percentfree (No valid datapoints found) [09:35:39] GoranSM: We allow user dbs to be created via ToolsDB for tools users - we don't really create mysql credentials for other labs instances. If you have are a tools user, you can use your replica.my.cnf from /home to create a database [09:36:02] (if you are not a tools user, we can add you) [09:36:24] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#User_databases [09:37:59] the databases on toolsdb are not backed up though, although we are working on offering some solutions in that area in the near future [10:04:19] madhuvishy: Thank you, got it now. [10:06:32] GoranSM: okay :) Let me know if you run into any trouble with the credentials or anything! [10:06:48] madhuvishy: Only one thing: I have copied the replica.my.cnf from a tool to my Labs instance (bd808 has suggested this). Question: where to do I connect in order to create a new database? [10:07:22] madhuvishy: I mean, it's not going to be -h enwiki.labsdb enwiki_p, neither -h enwiki.labsdb enwiki_p... where? [10:07:43] GoranSM: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#User_databases has steps to do that [10:07:50] mysql --defaults-file=$HOME/replica.my.cnf -h tools.labsdb [10:07:55] and then CREATE [10:08:00] madhuvishy: [10:08:02] mad [10:08:15] madhuvishy: Sorry. Got it, Ok. I'm on it. Thank you so much! [10:08:35] np :) [10:08:39] madhuvishy: :) [12:51:56] *waves* [12:53:22] Hello addshore [14:29:09] Can I install Open Web Analytics or Piwik for tools on Labs? Demo: http://demo.piwik.org Apparently WM-DE uses it. [15:18:59] Dispenser: There's no reason you couldn't use those tools in a VM. I don't know about installing them on tools exec hosts but you could certainly run them locally or in a venv. [15:19:16] If you need them installed on e.g. all exec nodes probably best to open a phabricator ticket so we can research [15:20:04] I'm asking because there's a vocal minority going "privacy" [15:20:44] ...whenever they find anything about our analytics [15:21:07] Hm... [15:21:17] It would certainly depend on what your analyzing and publishing [15:21:27] But that's not an objection against the software itself :) [15:26:05] "How can you tell I've used it on mobile? Is this being recorded/reported somewhere? I'm a bit unnerved at the thought of that." The first place winner in a contest bot monitored. [15:27:49] So heat maps and screen recorders might be more unnerving to people [15:37:44] It's hard for me to make a call on the spot… probably best to start a discussion on phab (or someplace) with details about what you want to do. Unless you've already done that? Seems like you're quoting an existing discussion [16:28:58] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:08:57] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [20:26:14] 06Labs, 10Labs-Infrastructure: Can't clone from some labs-instances - https://phabricator.wikimedia.org/T166170#3287386 (10Luke081515) [20:26:58] 06Labs, 10Labs-Infrastructure: Can't do git clone from some labs-instances - https://phabricator.wikimedia.org/T166170#3287398 (10Luke081515) [20:55:05] 10Tool-Labs-tools-Pageviews: Add namespace filter to Massviews - https://phabricator.wikimedia.org/T166176#3287517 (10MusikAnimal) [20:56:50] 10Tool-Labs-tools-Pageviews: Make options sticky in list-based apps - https://phabricator.wikimedia.org/T166178#3287549 (10MusikAnimal) [21:00:12] Dispenser: you could, but you won't love the result. You will see all IPs as one of the nginx proxies [21:01:18] Well screen resolution is more important for my use cases [21:06:18] the Foundation does use Piwik for analytics on blog.wikimedia.org and some other things [21:06:57] officially User-Agent is PII by the policy published on ... foundationwiki? [21:07:46] if we setup Piwik I think I'd like it to be done labs wide and then password protected somehow [21:08:42] its certainly worth a ticket to talk about. Chase and I were talking about the long standing need for some kind of per-tool hit counters about an hour ago while we ate lunch. :) [21:09:01] There's different between "Internet Explorer" and "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; YPC 3.2.0; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" [21:09:19] agreed [21:10:00] we publish browser family and version data for all of the prod wikis, so doing that for tools/labs seems ok [21:10:24] 10Tool-Labs-tools-Pageviews: Investigation: Recursive category search in Massviews - https://phabricator.wikimedia.org/T149334#3287664 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal Deployed! The depth is limited to 50 categories deep to prevent the code from going into an infinite loop, but this should... [21:11:08] there are some protections in the data for all wikis though too. The analytics team filters out things that are too specific by virtue of having too little traffic. [21:11:44] Like on the Amish WP (pdc.wikipedia.org) [21:12:30] yeah. I'm looking for their description of the filtering... [21:13:07] this is the reporting UI for prod wikis -- https://browser-reports.wmflabs.org/ [21:13:43] "For privacy and readability reasons, all the rows that initially represent less than K% of the total view count are grouped into a single row where all dimension columns (os_family, os_major, browser_family, browser_major) are assigned an unknown value (see job config for exact K and unknown values)." [21:14:44] so ... 0.05%? -- https://github.com/wikimedia/analytics-refinery/blob/master/oozie/browser/general/coordinator.properties#L61-L62 [21:16:06] there's a ticket somewhere in phab about figuring out how to feed the traffic logs for tools/labs into the main hadoop cluster. That would be pretty nice. [21:16:39] in prod that is all handled by things that take data from Varnish and we don't have that layer [21:16:48] so it needs some new data feed process [21:17:22] Or I guess we *could* add a varnish layer that just did hit-for-pass on everything... [21:17:45] hit-for-pass is varnish config speak for "ask the backend server"