[04:19:39] 10Quarry: Add 'download as wikitable' option (Quarry) - https://phabricator.wikimedia.org/T117645#2212584 (10zhuyifei1999) a:03zhuyifei1999 I was looking for this for ages. [09:17:45] 06Labs, 07Graphite: Setup "official labs grafana" instance - https://phabricator.wikimedia.org/T120295#2212726 (10Addshore) Also my primary case for a labs instance would be to test out new datasource plugins! Which I guess we are not okay doing on the production version. [11:04:13] 10Quarry: Show last run time on query page in Quarry - https://phabricator.wikimedia.org/T118646#2212761 (10XXN) [11:04:17] 10Quarry: Add date when query was last run - https://phabricator.wikimedia.org/T77941#2212762 (10XXN) [11:10:48] (03PS1) 10Luke081515: Display messages from mediawiki/core at #wikimedia-codereview [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/283850 (https://phabricator.wikimedia.org/T128371) [14:44:49] hello - anyone able to explain something to me about the labs copies of the production databases? [14:45:18] thparkth_: just ask the question :-) [14:45:28] the view we have of "page" doesn't seem to have an underlying index on page_title [14:45:36] is that intentional? [14:45:55] It does. [14:45:57] e.g select page_id from page where page_title="Elderslie_Rock"; [14:46:03] I would expect that to be instantaneous [14:46:06] it isn't :) [14:46:19] right, because the index is on (page_namespace,page_title) and not on just page_title [14:46:48] fair enough [14:46:57] "and page_namespace=0" is indeed immediate [14:47:00] I'll shut up now ;) [14:47:14] https://www.mediawiki.org/wiki/Manual:Page_table#Indices [14:47:34] there are a few subtleties in some tables on the replicas due to filtering [14:48:03] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#Tables_for_revision_or_logging_queries_involving_user_names_and_IDs [14:48:18] yup I've come accross those before [15:49:28] 06Labs, 10Tool-Labs: Web service failing to start - https://phabricator.wikimedia.org/T132877#2212900 (10Nettrom) [15:58:00] 06Labs, 10Tool-Labs: tools-webgrid-lighttpd-1415 disabled - https://phabricator.wikimedia.org/T132878#2212916 (10valhallasw) [15:59:29] 06Labs, 10Tool-Labs: Web service failing to start - https://phabricator.wikimedia.org/T132877#2212900 (10valhallasw) So, to summarize, 1) tools-webgrid-lighttpd-1401, -1411, -1412, -1413, -1414 are overloaded. This is still the case, 2) tools-webgrid-lighttpd-1415 is disabled ({T132878}) 3) all other hosts re... [16:03:01] 06Labs, 10Tool-Labs: High load on some webgrid nodes - https://phabricator.wikimedia.org/T132879#2212933 (10valhallasw) [16:18:00] 06Labs, 10Tool-Labs: High load on some webgrid nodes - https://phabricator.wikimedia.org/T132879#2212951 (10valhallasw) ``` for i in {01..14}; do echo tools-webgrid-lighttpd-14$i; echo "---------------------------------------"; ssh tools-webgrid-lighttpd-14$i 'top -b -n 2 -d 3 -S -o "-%CPU" | tail | t... [16:23:37] 06Labs, 10Tool-Labs: High load on some webgrid nodes - https://phabricator.wikimedia.org/T132879#2212953 (10valhallasw) From `strace`, the jobs seem to be in a busy-wait loop: ``` gettimeofday({1460909928, 640933}, NULL) = 0 gettimeofday({1460909928, 640985}, NULL) = 0 gettimeofday({1460909928, 641043}, NULL)... [16:28:16] 06Labs, 10Tool-Labs: High load on some webgrid nodes - https://phabricator.wikimedia.org/T132879#2212957 (10valhallasw) 05Open>03Resolved a:03valhallasw Cleaned up with ``` valhallasw@tools-bastion-03:~$ sudo become jembot tools.jembot@tools-bastion-03:~$ for i in {01..14}; do ssh tools-webgrid-lighttpd-... [16:29:02] 10Tool-Labs-tools-Other: tools.jembot spawns many php-cgi processes in busy loop (?) - https://phabricator.wikimedia.org/T132880#2212960 (10valhallasw) [17:20:53] PROBLEM - Host tools-worker-1011 is DOWN: PING CRITICAL - Packet loss = 100% [20:11:22] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [22:46:58] Another database question if anyone is feeling tolerant: How can this (https://en.wikipedia.org/w/index.php?title=Special%3ALog&type=delete&user=&page=Lakaz&year=&month=-1&tagfilter=&hide_thanks_log=1&hide_patrol_log=1&hide_tag_log=1&hide_review_log=1) return instantly, but this (select * from logging where log_type="delete" and log_action="delete" and log_namespace=0 and log_title="Lakaz";) wants to scan 24 million rows? [22:52:05] thparkth: try selecting from logging_logindex [22:52:49] yup that does it [22:53:04] in retrospect I suppose it's obvious I should have tried that :) [22:53:44] thanks Earwig