[07:47:55] <wikibugs>	 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1447936 (10Steinsplitter) >>! In T105585#1447296, @jcrespo wrote: > labsdb1002 crashed yesterday at 2:38 UTC due to excessive memory usage. I've restarted replication, there is not much to do now...
[07:48:30] <Steinsplitter>	 there is still +45998 replag on commonswiki_p.....
[09:26:07] <Betacommand>	 replag is kinda high still http://tools.wmflabs.org/betacommand-dev/cgi-bin/replag
[09:35:03] <Nemo_bis>	 I noticed too on tools I was using
[09:55:56] <shinken-wm>	 PROBLEM - Puppet failure on tools-static-02 is CRITICAL 30.00% of data above the critical threshold [0.0]
[10:07:33] <shinken-wm>	 PROBLEM - Puppet failure on tools-static-01 is CRITICAL 50.00% of data above the critical threshold [0.0]
[10:08:16] <Superyetkin>	 hello
[10:08:31] <Superyetkin>	 what happened to user databases on c2?
[10:09:32] <Superyetkin>	 anyone there?
[10:21:04] <Superyetkin>	 hey
[10:43:24] <Superyetkin>	 anyone there?
[10:43:44] <Superyetkin>	 all c's seem to be lagging in replication
[11:38:25] <Superyetkin>	 anyone there to  respond?
[11:45:31] <Nemo_bis>	 Superyetkin: AFAIK we're slowly recovering from a replag issue, which should be getting better since some 24 hours ago
[11:48:02] <Superyetkin>	 Nemo_bis: ok, but where is my user database?
[11:48:25] <Superyetkin>	 see http://tools.wmflabs.org/superyetkin/istenen_ogeler.php
[11:53:30] <Superyetkin>	 table does not exist?
[11:56:57] <wikibugs>	 10Wikibugs: Wikibugs should not send IRC notifications if the only change was CC - https://phabricator.wikimedia.org/T105625#1448041 (10Krinkle) 3NEW
[12:05:21] <Superyetkin>	 anyone who can explain why user databases were broken?
[13:23:12] <zhuyifei1999_>	 !log commonsarchive Enabled HSTS
[14:15:48] <wikibugs>	 6Labs, 10Labs-Infrastructure: wikidatawiki_p on Labs lags ~20h behind live site - https://phabricator.wikimedia.org/T105632#1448146 (10Magnus) 3NEW
[14:19:24] <Huji>	 Requesting help: when I try to add a rule to a security group that I created myself, I get the "Failed to add rule" error.
[14:42:29] <wikibugs>	 10Tool-Labs-tools-Other: Some geolocated photos not shown - https://phabricator.wikimedia.org/T105593#1448196 (10Aklapper)
[14:51:30] <Nemo_bis>	 Betacommand: seems stalled again, now over 70k https://tools.wmflabs.org/betacommand-dev/cgi-bin/replag
[14:56:15] <Betacommand>	 Nemo_bis: i saw, nothing I can do though
[14:56:22] <Nemo_bis>	 sure
[15:34:26] <wikibugs>	 6Labs, 10Adminbot, 6operations: Make morebots run on a production host - https://phabricator.wikimedia.org/T94638#1448264 (10Krinkle)
[18:50:26] <Huji>	 Hi. Can someone please help me with my question regarding security groups?
[18:50:40] <Huji>	 If so, please ping me in your reply.
[19:01:58] <wikibugs>	 10Wikibugs: Wikibugs should not send IRC notifications if the only change was CC - https://phabricator.wikimedia.org/T105625#1448401 (10Legoktm) ``` 2015-07-12 11:51:18,623 - wikibugs.wb2-phab - DEBUG - get_transaction_info(104164,OrderedDict([('PHID-XACT-TASK-wb3glbgrmvzsfpf', 'PHID-XACT-TASK-wb3glbgrmvzsfpf'),...
[19:03:27] <wikibugs>	 10Wikibugs: Wikibugs should not send IRC notifications if the only change was CC - https://phabricator.wikimedia.org/T105625#1448403 (10Legoktm)
[19:17:01] <wikibugs>	 10Wikibugs: Wikibugs should not send IRC notifications if the only change was CC - https://phabricator.wikimedia.org/T105625#1448437 (10Legoktm) the phab API (manifest.gettasktransactions) exposes only the following data: ```     {       "taskID": "104164",       "transactionPHID": "PHID-XACT-TASK-2bgwvcst6bllpu...
[19:19:52] <wikibugs>	 10Wikibugs: Wikibugs should ignore changes to the security field - https://phabricator.wikimedia.org/T105625#1448439 (10Legoktm)
[21:14:29] <wikibugs>	 6Labs, 10Tool-Labs, 7Database: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1448504 (10Nemo_bis)
[21:14:56] <wikibugs>	 6Labs, 10Tool-Labs, 7Database: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1447265 (10Nemo_bis)
[22:22:16] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Hardwigg was created, changed by Hardwigg link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Hardwigg edit summary: Created page with "{{Tools Access Request |Justification=To host projects that access Wikidata's data more than 50 at a time. Here's a project I've created on Github: https://github.com/cdrini/w..."
[22:58:06] <wikibugs>	 6Labs, 10Tool-Labs, 7Database: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1448573 (10Betacommand) Regardless with what happened to the primary mariadb server (wmf switched to this a while back) the actual database replication is still non-functional. As of...