[04:20:43] 10DBA, 10Phabricator, 10Release-Engineering-Team-TODO, 10Documentation, and 2 others: Prepare a disaster recovery plan for failing over Phabricator - https://phabricator.wikimedia.org/T190572 (10Dzahn) [06:15:02] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [06:32:40] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [07:09:58] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [09:03:20] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [12:12:35] 10DBA, 10GlobalUsage, 10StructuredDataOnCommons: Normalize globalimagelinks table - https://phabricator.wikimedia.org/T241053 (10Ladsgroup) I add @daniel and @Fjalapeno from CPT to take a look at this. Maybe it can be prioritized and done easily. [13:12:37] 10DBA, 10Dumps-Generation, 10MW-1.35-notes (1.35.0-wmf.10; 2019-12-10): Some mw snapshot hosts are accessing main db servers - https://phabricator.wikimedia.org/T143870 (10Marostegui) I have quickly checked and so far I have seen the following on s2 I see connections (sleeping) from snapshoot1006 and 1007 t... [14:06:20] 10DBA, 10Dumps-Generation, 10MW-1.35-notes (1.35.0-wmf.10; 2019-12-10): Some mw snapshot hosts are accessing main db servers - https://phabricator.wikimedia.org/T143870 (10ArielGlenn) Vslow should have all of them, so there may be some last place that short queries or just connections happen. I'll take anoth... [14:22:12] 10DBA, 10Core Platform Team, 10GlobalUsage, 10StructuredDataOnCommons: Normalize globalimagelinks table - https://phabricator.wikimedia.org/T241053 (10daniel) Tagging #core_platform_team for triage. This is not a quick job, it's a technical infrastructure project. Not a big one, but a project in its own r... [15:10:55] 10DBA: db1130 BBU possible issues - https://phabricator.wikimedia.org/T240823 (10Marostegui) >>! In T240823#5746805, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=https://tools.wmflabs.org/sal/log/AW8SdviHvrJzePItvmHv} [2019-12-17T06:06:24Z] Upgrade db1130 T2... [16:10:46] 10DBA, 10Core Platform Team, 10GlobalUsage, 10StructuredDataOnCommons: Normalize globalimagelinks table - https://phabricator.wikimedia.org/T241053 (10Anomie) >>! In T241053#5757157, @daniel wrote: > gil_page_namespace could too maybe, or it could be the namespace ID - the actual namespace name would have... [21:45:05] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1123 - https://phabricator.wikimedia.org/T240534 (10Jclark-ctr) @marostegui disk arrived today message me on irc if available to change [22:59:49] 10DBA, 10Epic: Setup es4 and es5 replica sets for new read-write external store service - https://phabricator.wikimedia.org/T226704 (10Jclark-ctr)