[09:02:38] 10DBA, 10Data-Services: Prepare and check storage layer for eowikivoyage - https://phabricator.wikimedia.org/T269427 (10LSobanski) Thanks, let us know when the database is created, so we can sanitize it. [09:03:05] 10DBA, 10Data-Services: Prepare and check storage layer for wawikisource - https://phabricator.wikimedia.org/T269432 (10LSobanski) Thanks, let us know when the database is created, so we can sanitize it. [09:06:39] 10DBA, 10Data-Services: Prepare and check storage layer for madwiki - https://phabricator.wikimedia.org/T269440 (10LSobanski) Thanks, let us know when the database is created, so we can sanitize it. [09:57:29] 10DBA, 10MediaWiki-General, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Patch-For-Review, and 2 others: Normalise MW Core database language fields length - https://phabricator.wikimedia.org/T253276 (10LSobanski) 05Resolved→03Open a:05Marostegui→03None Reopening based on @Ladsgroup's comment. [11:32:20] 10DBA, 10Data-Services: Prepare and check storage layer for madwiki - https://phabricator.wikimedia.org/T269440 (10LSobanski) p:05Triage→03Medium [11:32:28] 10DBA, 10Data-Services: Prepare and check storage layer for wawikisource - https://phabricator.wikimedia.org/T269432 (10LSobanski) p:05Triage→03Medium [11:32:35] 10DBA, 10Data-Services: Prepare and check storage layer for eowikivoyage - https://phabricator.wikimedia.org/T269427 (10LSobanski) p:05Triage→03Medium [11:36:47] 10DBA: transfer.py fails when copying data between es hosts - https://phabricator.wikimedia.org/T262388 (10LSobanski) As discussed last week, this task needs to be estimated and scheduled. It's not currently considered to be a workflow blocker. [18:30:35] 10DBA, 10GrowthExperiments, 10Growth-Team (Current Sprint), 10Patch-For-Review, and 2 others: Slow load times for Special:Homepage on cswiki - https://phabricator.wikimedia.org/T267216 (10MMiller_WMF) @Tgr has an idea to use ElasticSearch for this, instead of a database query. More information will be for... [19:21:25] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Deploy labsdbuser and views to new clouddb hosts - https://phabricator.wikimedia.org/T268312 (10Bstorm) >>! In T268312#6669069, @Marostegui wrote: > @Bstorm clouddb1020:3315 and clouddb1020:3318 is now up. > Can you double check th... [19:21:41] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Deploy labsdbuser and views to new clouddb hosts - https://phabricator.wikimedia.org/T268312 (10Bstorm) [19:23:25] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Deploy labsdbuser and views to new clouddb hosts - https://phabricator.wikimedia.org/T268312 (10Bstorm) 05Open→03Resolved [19:23:31] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Bstorm) [22:41:12] PROBLEM - MariaDB sustained replica lag on db1084 is CRITICAL: 3.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1084&var-port=9104 [22:44:14] PROBLEM - MariaDB sustained replica lag on db1118 is CRITICAL: 8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1118&var-port=9104 [22:46:12] PROBLEM - MariaDB sustained replica lag on db1106 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1106&var-port=9104 [22:46:30] PROBLEM - MariaDB sustained replica lag on db1135 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1135&var-port=9104 [22:48:02] PROBLEM - MariaDB sustained replica lag on db1089 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1089&var-port=9104 [22:48:04] RECOVERY - MariaDB sustained replica lag on db1135 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1135&var-port=9104 [22:48:28] PROBLEM - MariaDB sustained replica lag on db2112 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2112&var-port=9104 [22:48:42] PROBLEM - MariaDB sustained replica lag on db1119 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1119&var-port=9104 [22:49:22] PROBLEM - MariaDB sustained replica lag on db1106 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1106&var-port=9104 [22:50:10] PROBLEM - MariaDB sustained replica lag on db1134 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1134&var-port=9104 [22:50:34] PROBLEM - MariaDB sustained replica lag on db1118 is CRITICAL: 3.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1118&var-port=9104 [22:50:44] PROBLEM - MariaDB sustained replica lag on db2072 is CRITICAL: 8.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2072&var-port=9104 [22:54:12] RECOVERY - MariaDB sustained replica lag on db1106 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1106&var-port=9104 [22:54:54] RECOVERY - MariaDB sustained replica lag on db2112 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2112&var-port=9104 [22:55:00] RECOVERY - MariaDB sustained replica lag on db1134 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1134&var-port=9104 [22:55:08] RECOVERY - MariaDB sustained replica lag on db1119 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1119&var-port=9104 [22:55:24] RECOVERY - MariaDB sustained replica lag on db1118 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1118&var-port=9104 [22:55:28] RECOVERY - MariaDB sustained replica lag on db1084 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1084&var-port=9104 [22:55:34] RECOVERY - MariaDB sustained replica lag on db2072 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2072&var-port=9104 [22:56:04] RECOVERY - MariaDB sustained replica lag on db1089 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1089&var-port=9104