[11:49:12] 10DBA, 10AbuseFilter, 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10Daimona) [11:49:30] 10DBA, 10AbuseFilter, 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10Daimona) [11:50:23] 10DBA, 10AbuseFilter, 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10Daimona) p:05Triage→03Unbreak! Train blockers are UBN, and this is a TB because it's impossible to open Special:AbuseLog or even use the API (plus hotfix coming). [11:52:03] 10DBA, 10AbuseFilter, 10Patch-For-Review, 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10Daimona) a:03Daimona [15:19:09] 10DBA, 10AbuseFilter, 10Patch-For-Review, 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10matej_suchanek) > Maybe a DBA could provide some explanation? Not a DBA but perhaps: because of the `LEFT JOIN`, the database starts scanning `... [16:35:19] 10DBA, 10AbuseFilter, 10MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), 10Patch-For-Review, 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10Daimona) @matej_suchanek Ah, so the index isn't used because of the cast? That's interesting, I'll... [16:47:13] 10DBA, 10AbuseFilter, 10MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), 10Patch-For-Review, 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10Daimona) >>! In T221357#5123211, @Daimona wrote: > @matej_suchanek Ah, so the index isn't used bec... [17:21:23] 10DBA, 10AbuseFilter, 10MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), 10Patch-For-Review, 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10mmodell) Synchronized php-1.34.0-wmf.1/extensions/AbuseFilter/includes/: sync https://gerrit.wikim... [17:22:16] 10DBA, 10AbuseFilter, 10MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), 10Patch-For-Review, 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10mmodell) [17:22:37] 10DBA, 10AbuseFilter, 10MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), 10Patch-For-Review, 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10mmodell) p:05Unbreak!→03Normal [17:22:47] 10DBA, 10AbuseFilter, 10MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), 10Patch-For-Review, 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10mmodell) No longer blocking the train, thanks. [17:26:55] 10DBA, 10AbuseFilter, 10MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), 10Patch-For-Review, 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10Daimona) Thanks @mmodell! Now we have plenty of time to understand how to get this working. Howeve... [17:29:41] 10DBA, 10AbuseFilter, 10MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), 10Patch-For-Review, 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10mmodell) @daimona: I don't know how important PostgreSQL support is to you, however, the normalize... [17:59:48] 10DBA, 10AbuseFilter, 10MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), 10Patch-For-Review, 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10matej_suchanek) [18:00:21] 10DBA, 10AbuseFilter, 10MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10matej_suchanek) [18:01:15] 10DBA, 10AbuseFilter, 10MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10Marostegui) @daimona if you let me know the query you'd like to try with the FORCE I can try to run it on a core host fo... [18:04:55] 10DBA, 10AbuseFilter, 10MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), 10Wikimedia-production-error: Read timeout reached while viewing AbuseLog - https://phabricator.wikimedia.org/T221357 (10Daimona) @mmodell Well, I guess we should always try to keep our code cross-DBMS-compat, and Postgres compatibility is m... [18:28:54] 10DBA, 10Operations, 10ops-eqiad, 10Goal, and 2 others: rack/setup/install db11[26-38].eqiad.wmnet - https://phabricator.wikimedia.org/T211613 (10Marostegui) p:05Normal→03High [18:31:54] 10DBA, 10MediaWiki-API, 10Patch-For-Review: Slow query "ApiQueryLogEvents::execute" after actor rollout - https://phabricator.wikimedia.org/T220999 (10Marostegui) p:05Triage→03Normal [20:49:21] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install db1139|db1140.eqiad.wmnet (2 dump slaves) - https://phabricator.wikimedia.org/T218985 (10RobH) [20:50:15] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install db1139|db1140.eqiad.wmnet (2 dump slaves) - https://phabricator.wikimedia.org/T218985 (10RobH) [20:51:55] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T219399 (10RobH) [21:07:38] 10DBA, 10Operations, 10Patch-For-Review: correctable memory errors db1068 (commons primary master database) - https://phabricator.wikimedia.org/T213664 (10Dzahn) 05Resolved→03Open https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=db1068&service=EDAC+syslog+messages https://icinga.wikime... [22:33:42] 10DBA, 10DC-Ops, 10decommission: decommission db2014,db2020, db2021, db2022, db2024, db2031 - https://phabricator.wikimedia.org/T221424 (10RobH) [22:34:49] 10DBA, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission db2014,db2020, db2021, db2022, db2024, db2031 - https://phabricator.wikimedia.org/T221424 (10RobH)