[00:08:21] RECOVERY - MariaDB sustained replica lag on db1081 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [00:32:35] PROBLEM - MariaDB sustained replica lag on db1081 is CRITICAL: 10.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [00:34:01] PROBLEM - MariaDB sustained replica lag on db1149 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1149&var-port=9104 [00:35:25] PROBLEM - MariaDB sustained replica lag on db1121 is CRITICAL: 3.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [00:35:45] RECOVERY - MariaDB sustained replica lag on db1149 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1149&var-port=9104 [00:38:53] RECOVERY - MariaDB sustained replica lag on db1121 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [00:45:09] PROBLEM - MariaDB sustained replica lag on db1148 is CRITICAL: 7.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1148&var-port=9104 [00:50:33] RECOVERY - MariaDB sustained replica lag on db1148 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1148&var-port=9104 [01:00:21] PROBLEM - MariaDB sustained replica lag on db1147 is CRITICAL: 11.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1147&var-port=9104 [01:00:57] PROBLEM - MariaDB sustained replica lag on db1148 is CRITICAL: 13.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1148&var-port=9104 [01:03:49] RECOVERY - MariaDB sustained replica lag on db1147 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1147&var-port=9104 [01:06:11] RECOVERY - MariaDB sustained replica lag on db1148 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1148&var-port=9104 [01:08:07] PROBLEM - MariaDB sustained replica lag on db1142 is CRITICAL: 7.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1142&var-port=9104 [01:09:49] RECOVERY - MariaDB sustained replica lag on db1142 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1142&var-port=9104 [01:21:11] PROBLEM - MariaDB sustained replica lag on db1147 is CRITICAL: 4.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1147&var-port=9104 [01:21:59] PROBLEM - MariaDB sustained replica lag on db1142 is CRITICAL: 5.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1142&var-port=9104 [01:22:55] RECOVERY - MariaDB sustained replica lag on db1147 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1147&var-port=9104 [01:23:43] RECOVERY - MariaDB sustained replica lag on db1142 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1142&var-port=9104 [01:28:43] RECOVERY - MariaDB sustained replica lag on db1081 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [06:26:50] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) Double checked that replication is enabled on all masters (on both dcs) [06:33:44] 10DBA: Orchestrator: Create basic documentation - https://phabricator.wikimedia.org/T266428 (10Marostegui) [06:34:16] 10DBA: Orchestrator: Create basic documentation - https://phabricator.wikimedia.org/T266428 (10Marostegui) p:05Triage→03Low We are far from having it in production, so this is not urgent at the moment. [06:36:00] 10DBA, 10Operations, 10Patch-For-Review, 10User-Kormat: orchestrator: Add service monitoring - https://phabricator.wikimedia.org/T266338 (10Marostegui) p:05Triage→03Low We don't have it in production, so putting this to low as we aren't on a hurry for this as of today [07:19:34] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) [07:48:01] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) [07:52:57] PROBLEM - MariaDB sustained replica lag on db1081 is CRITICAL: 27.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [07:54:03] PROBLEM - MariaDB sustained replica lag on db1121 is CRITICAL: 5.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [07:57:31] RECOVERY - MariaDB sustained replica lag on db1121 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [08:03:33] PROBLEM - MariaDB sustained replica lag on db1142 is CRITICAL: 8.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1142&var-port=9104 [08:05:17] RECOVERY - MariaDB sustained replica lag on db1142 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1142&var-port=9104 [08:08:53] PROBLEM - MariaDB sustained replica lag on db1146 is CRITICAL: 15.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1146&var-port=13314 [08:11:25] PROBLEM - MariaDB sustained replica lag on db1121 is CRITICAL: 16.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [08:11:27] 10DBA, 10Commons, 10Operations, 10Release-Engineering-Team: Increase on database writes and deletes activity on Commonswiki lead to some replication lag - https://phabricator.wikimedia.org/T266432 (10Marostegui) [08:11:37] 10DBA, 10Commons, 10Operations, 10Release-Engineering-Team: Increase on database writes and deletes activity on Commonswiki lead to some replication lag - https://phabricator.wikimedia.org/T266432 (10Marostegui) p:05Triage→03Medium [08:12:06] 10DBA, 10Commons, 10Operations, 10Release-Engineering-Team: Increase on database writes and deletes activity on Commonswiki lead to some replication lag - https://phabricator.wikimedia.org/T266432 (10Marostegui) p:05Medium→03High Setting to high as this might be causing cross dc lag [08:12:27] 10DBA, 10Commons, 10Operations, 10Release-Engineering-Team: Increase on database writes and deletes activity on Commonswiki leads to some replication lag - https://phabricator.wikimedia.org/T266432 (10Marostegui) [08:13:09] RECOVERY - MariaDB sustained replica lag on db1121 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [08:17:19] PROBLEM - MariaDB sustained replica lag on db1143 is CRITICAL: 19.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1143&var-port=9104 [08:18:43] PROBLEM - MariaDB sustained replica lag on db1149 is CRITICAL: 35.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1149&var-port=9104 [08:19:01] PROBLEM - MariaDB sustained replica lag on db1148 is CRITICAL: 17.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1148&var-port=9104 [08:19:03] RECOVERY - MariaDB sustained replica lag on db1143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1143&var-port=9104 [08:19:17] RECOVERY - MariaDB sustained replica lag on db1146 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1146&var-port=13314 [08:22:13] RECOVERY - MariaDB sustained replica lag on db1149 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1149&var-port=9104 [08:25:51] RECOVERY - MariaDB sustained replica lag on db1148 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1148&var-port=9104 [08:45:50] o/ [09:10:31] RECOVERY - MariaDB sustained replica lag on db1081 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [10:27:42] 10DBA, 10Operations, 10User-Kormat: Integrate orchestrator with !log - https://phabricator.wikimedia.org/T266452 (10Marostegui) [10:27:54] 10DBA, 10Operations, 10User-Kormat: Integrate orchestrator with !log - https://phabricator.wikimedia.org/T266452 (10Marostegui) p:05Triage→03Medium [10:28:33] sobanski: morning :) [10:28:45] has anyone looked into the sustained lag warnings over the weekend? [10:29:15] Morning! I certainly didn't :( [10:29:30] kormat: I did, I have created a task about it [10:29:49] https://phabricator.wikimedia.org/T266432 [10:30:09] ah, great :) [10:30:18] There were also some issues with someone uploading a bunch of videos to common today, which might have helped to increase the rate of alerting in the last few hours [10:36:19] xmldumps started today at around 6UTC, right? [10:36:36] don't know [10:36:38] https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&var-site=eqiad&var-group=core&var-shard=All&var-role=All&from=1603686993635&to=1603708593636 [10:36:57] I meant: https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&var-site=eqiad&var-group=core&var-shard=All&var-role=All&from=1603686993635&to=1603708593636&viewPanel=8 [10:39:25] that is me warming up tables [10:39:30] ah! [10:39:36] cool then :-) [10:40:17] Hopefully not for long, with all the warming up going on ;) [10:40:44] * sobanski will see himself out [10:50:00] 10DBA, 10Commons, 10Operations, 10Release-Engineering-Team, 10Wikimedia-production-error: Increase on database writes and deletes activity on Commonswiki leads to some replication lag - https://phabricator.wikimedia.org/T266432 (10jcrespo) Adding #Wikimedia-production-error as it seems to coincide with a... [12:24:23] 10DBA, 10Commons, 10Operations, 10Release-Engineering-Team, 10Wikimedia-production-error: Increase on database writes and deletes activity on Commonswiki leads to some replication lag - https://phabricator.wikimedia.org/T266432 (10Marostegui) >>! In T266432#6577770, @jcrespo wrote: > Adding #Wikimedia-pr... [14:18:39] 10DBA, 10Commons, 10Operations, 10Release-Engineering-Team, 10Wikimedia-production-error: Increase on database writes and deletes activity on Commonswiki leads to some replication lag - https://phabricator.wikimedia.org/T266432 (10jcrespo) I am getting strange, inconsistent results every time I check, no... [14:31:42] 10DBA, 10Data-Persistence, 10Growth-Structured-Tasks, 10Growth-Team (Current Sprint): Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10kostajh) >>! In T261411#6573592, @Tgr wrote: > I think this is done, per the last two... [14:36:04] 10DBA, 10Commons, 10Operations, 10Release-Engineering-Team, 10Wikimedia-production-error: Increase on database writes and deletes activity on Commonswiki leads to some replication lag - https://phabricator.wikimedia.org/T266432 (10thcipriani) > Was something released that 22nd Oct? Commonswiki was updat... [14:36:19] 10DBA, 10wikitech.wikimedia.org: Move database for wikitech (labswiki) to a main cluster section - https://phabricator.wikimedia.org/T167973 (10Andrew) Checking in -- could we go ahead and make this move after the datacenter switchover? [14:40:24] 10DBA, 10wikitech.wikimedia.org: Move database for wikitech (labswiki) to a main cluster section - https://phabricator.wikimedia.org/T167973 (10Marostegui) The DC switchover is tomorrow, so we can try to plan for it in Q2, if we find the time for it. I will ping you once I've come up with a plan! Thanks! [14:51:21] I think backup started 2 hours ago [14:51:29] *backups, in particular snapshots [14:53:56] did any of you start backup or are performing maintenance on s1 and s8 source backups? [14:55:59] nop [14:56:58] I think either an error happened or someone manually stopped replication on Oct 26 12:48:42 [14:57:14] possiblity with the backup script, as s1 and s8 stopped at the same time [14:57:27] which is exactly what the backup script starts with [14:59:26] 10DBA, 10Commons, 10Operations, 10Release-Engineering-Team, 10Wikimedia-production-error: Increase on database writes and deletes activity on Commonswiki leads to some replication lag - https://phabricator.wikimedia.org/T266432 (10thcipriani) Adding @LarsWirzenius in-case he remembers anything deploying... [15:04:20] so there is 99% probability it is just someone running backups accidentaly and then Ctr-c, but I want to be sure it is not something else [15:04:47] sadly, we have the backups running with logs disabled until I fix a bug [15:10:16] jynus: not sure who'd do that really [15:10:17] confirmed it is the backups, I can see the rests of the transfers but not preparing on dbprov hosts [15:10:47] which would fit an "accidental" execution of backups and ctrl-c on cumin, which doesn't really kill the remote processes [15:17:48] mystery found, I will restart replication and remove unprepared backups [15:18:01] will also work on reenabling logs asap [15:40:57] 10DBA, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Kormat) p:05Triage→03Medium [15:43:44] 10DBA, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10jcrespo) I believe there was a ticket were we refereed this, let me try to search it as I think I run into this issue for the WMFReplication class. [15:53:25] 10DBA, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [15:56:11] PROBLEM - MariaDB sustained replica lag on db1081 is CRITICAL: 26.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [15:56:15] PROBLEM - MariaDB sustained replica lag on db1149 is CRITICAL: 9.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1149&var-port=9104 [15:57:27] PROBLEM - MariaDB sustained replica lag on db1121 is CRITICAL: 5.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [15:57:58] 10DBA, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10jcrespo) I didn't find a ticket, so maybe it was only an informal conversation with no actionables. This was something we wanted to do, because when implementing "primaryhost.slaves()" on WMFMariaDB code we didn... [15:58:09] 10DBA: Populating orchestrator metadata on a per-server basis - https://phabricator.wikimedia.org/T266485 (10Marostegui) [15:58:27] 10DBA: Populating orchestrator metadata on a per-server basis - https://phabricator.wikimedia.org/T266485 (10Marostegui) p:05Triage→03Medium [15:59:12] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10Cmjohnson) [16:02:03] 10DBA: Populating orchestrator metadata on a per-server basis - https://phabricator.wikimedia.org/T266485 (10Marostegui) Maybe this can be placed on the `ops` database already. We'd need to deploy the following grants everywhere: ` GRANT SELECT ON ops.cluster TO 'orchestrator'@'orc_host'; ` [16:02:09] PROBLEM - MariaDB sustained replica lag on db1141 is CRITICAL: 3.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1141&var-port=9104 [16:02:54] 10DBA: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10LSobanski) [16:03:17] 10DBA, 10Growth-Structured-Tasks, 10Growth-Team (Current Sprint): Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10LSobanski) [16:03:29] 10DBA: Monitor the growth of CheckUser tables at large wikis - https://phabricator.wikimedia.org/T265344 (10LSobanski) [16:03:40] 10DBA: Evaluate the impact of changing innodb_change_buffering to inserts - https://phabricator.wikimedia.org/T263443 (10LSobanski) [16:04:07] RECOVERY - MariaDB sustained replica lag on db1149 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1149&var-port=9104 [16:04:55] 10Blocked-on-schema-change, 10DBA, 10Operations, 10User-Kormat: Schema change to make change_tag.ct_rc_id unsigned - https://phabricator.wikimedia.org/T259831 (10LSobanski) [16:05:17] RECOVERY - MariaDB sustained replica lag on db1121 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [16:06:59] 10DBA, 10Commons, 10Operations, 10Release-Engineering-Team, 10Wikimedia-production-error: Increase on database writes and deletes activity on Commonswiki leads to some replication lag - https://phabricator.wikimedia.org/T266432 (10Marostegui) We just had another huge spike of DELETEs {F32414787} [16:08:01] RECOVERY - MariaDB sustained replica lag on db1141 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1141&var-port=9104 [16:09:48] 10DBA, 10Operations, 10Patch-For-Review, 10User-Kormat, 10User-jbond: Refactor mariadb puppet code - https://phabricator.wikimedia.org/T256972 (10Kormat) [16:21:43] RECOVERY - MariaDB sustained replica lag on db1081 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [16:23:32] trivia: I run my file-downloader script, and then I counted the total number of images with "ls | wc -l" [16:23:42] I was missing one image, can you guess why? [16:24:58] jynus: not sure.. [16:25:13] ls doesn't show all images [16:25:23] can you guess which one I missed [16:25:28] ? [16:25:40] what do you mean by image in this context [16:25:41] ? [16:25:44] file [16:25:54] https://test.wikipedia.org/wiki/File:00000218.gif [16:26:07] ^example of one of the 4K images I downloaded to dbprov1003 [16:26:17] is.. there a file beginning with `.`? [16:26:20] yep [16:26:24] https://test.wikipedia.org/wiki/File:.scream.jpeg [16:26:31] :-D you got it [16:27:22] happily, images with ".." are banned from being uploaded [16:29:23] also sha1 doesn't identify uniquely an image [16:29:31] 10DBA, 10Commons, 10Operations, 10Release-Engineering-Team, 10Wikimedia-production-error: Increase on database writes and deletes activity on Commonswiki leads to some replication lag - https://phabricator.wikimedia.org/T266432 (10LarsWirzenius) @thcipriani Sorry, I have no recollection that anything tha... [16:29:47] I mean, it does, but the same image can be uploaded more than once [16:29:57] under different names [16:30:06] all things I am learning these weeks [16:32:50] ahh [16:33:25] there is one more thing I want to show you, which could be useful for database administration: sha1 is encoded on the db on base36, so if you query the image table and sha1 they won't apparently match... because one will return a hex number and the other a base36 one [16:36:07] so I think if I need a unique file identifier I will need wiki + name + sha1 (maybe + timestamp, depending on my definition of file) [16:37:22] 10Blocked-on-schema-change: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Ladsgroup) [16:37:40] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Ladsgroup) [16:39:13] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) p:05Triage→03Medium [16:57:07] PROBLEM - MariaDB sustained replica lag on db1081 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [16:58:57] RECOVERY - MariaDB sustained replica lag on db1081 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [17:19:45] PROBLEM - MariaDB sustained replica lag on db1081 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [17:24:03] 10DBA: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) [17:24:07] 10DBA: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) 05Open→03Resolved This is all done [17:24:23] PROBLEM - MariaDB sustained replica lag on db1081 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [17:33:03] RECOVERY - MariaDB sustained replica lag on db1081 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [17:40:02] 10DBA, 10Operations, 10User-Kormat: orchestrator: Add service monitoring - https://phabricator.wikimedia.org/T266338 (10Dzahn) New checks have been added to Icinga: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=orchestrator But notifications for everything on this new host are disabl... [17:53:33] PROBLEM - MariaDB sustained replica lag on db1081 is CRITICAL: 12 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [17:56:43] PROBLEM - MariaDB sustained replica lag on db1143 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1143&var-port=9104 [17:58:31] RECOVERY - MariaDB sustained replica lag on db1143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1143&var-port=9104 [18:09:29] PROBLEM - MariaDB sustained replica lag on db1141 is CRITICAL: 6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1141&var-port=9104 [18:10:11] PROBLEM - MariaDB sustained replica lag on db1147 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1147&var-port=9104 [18:11:15] RECOVERY - MariaDB sustained replica lag on db1141 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1141&var-port=9104 [18:11:59] RECOVERY - MariaDB sustained replica lag on db1147 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1147&var-port=9104 [18:13:13] PROBLEM - MariaDB sustained replica lag on db1138 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1138&var-port=9104 [18:18:13] PROBLEM - MariaDB sustained replica lag on db1146 is CRITICAL: 8.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1146&var-port=13314 [18:19:14] 10DBA, 10Commons, 10Operations, 10Platform Engineering, and 2 others: Increase on database writes and deletes activity on Commonswiki leads to some replication lag - https://phabricator.wikimedia.org/T266432 (10thcipriani) Here is the changelog of all patchsets that went out last week: https://www.mediaw... [18:20:03] PROBLEM - MariaDB sustained replica lag on db1141 is CRITICAL: 4.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1141&var-port=9104 [18:20:17] RECOVERY - MariaDB sustained replica lag on db1138 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1138&var-port=9104 [18:23:31] RECOVERY - MariaDB sustained replica lag on db1146 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1146&var-port=13314 [18:28:55] RECOVERY - MariaDB sustained replica lag on db1141 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1141&var-port=9104 [18:30:39] RECOVERY - MariaDB sustained replica lag on db1081 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [18:41:41] 10DBA: Monitor the growth of CheckUser tables at large wikis - https://phabricator.wikimedia.org/T265344 (10Huji) @Marostegui let's say you pull data tomorrow, and you see the current trends continue; by current trends, I mean the uncompressed size increasing by 14MB a week for eswiki, and 20MB for ruwiki. What... [21:53:43] PROBLEM - MariaDB sustained replica lag on db1081 is CRITICAL: 12.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1081&var-port=9104 [21:55:41] PROBLEM - MariaDB sustained replica lag on db1149 is CRITICAL: 4.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1149&var-port=9104 [22:01:03] RECOVERY - MariaDB sustained replica lag on db1149 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1149&var-port=9104 [22:01:37] PROBLEM - MariaDB sustained replica lag on db1147 is CRITICAL: 4.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1147&var-port=9104 [22:03:59] PROBLEM - MariaDB sustained replica lag on db1148 is CRITICAL: 4.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1148&var-port=9104 [22:05:13] RECOVERY - MariaDB sustained replica lag on db1147 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1147&var-port=9104 [22:09:25] RECOVERY - MariaDB sustained replica lag on db1148 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1148&var-port=9104 [22:09:57] PROBLEM - MariaDB sustained replica lag on db1141 is CRITICAL: 63 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1141&var-port=9104 [22:11:41] PROBLEM - MariaDB sustained replica lag on db1146 is CRITICAL: 35.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1146&var-port=13314 [22:18:55] RECOVERY - MariaDB sustained replica lag on db1146 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1146&var-port=13314 [22:18:59] RECOVERY - MariaDB sustained replica lag on db1141 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1141&var-port=9104 [22:19:37] PROBLEM - MariaDB sustained replica lag on db1142 is CRITICAL: 54.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1142&var-port=9104 [22:22:59] PROBLEM - MariaDB sustained replica lag on db1121 is CRITICAL: 69.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [22:24:59] RECOVERY - MariaDB sustained replica lag on db1142 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1142&var-port=9104 [22:27:55] PROBLEM - MariaDB sustained replica lag on db1146 is CRITICAL: 137.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1146&var-port=13314 [22:27:59] PROBLEM - MariaDB sustained replica lag on db1141 is CRITICAL: 137.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1141&var-port=9104 [22:28:23] RECOVERY - MariaDB sustained replica lag on db1121 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [22:29:43] RECOVERY - MariaDB sustained replica lag on db1146 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1146&var-port=13314 [22:31:35] RECOVERY - MariaDB sustained replica lag on db1141 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1141&var-port=9104 [22:41:21] PROBLEM - MariaDB sustained replica lag on db1147 is CRITICAL: 61.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1147&var-port=9104 [22:44:59] RECOVERY - MariaDB sustained replica lag on db1147 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1147&var-port=9104 [22:46:29] PROBLEM - MariaDB sustained replica lag on db1121 is CRITICAL: 52.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [22:57:19] RECOVERY - MariaDB sustained replica lag on db1121 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [23:01:11] PROBLEM - MariaDB sustained replica lag on db1147 is CRITICAL: 86.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1147&var-port=9104 [23:06:39] RECOVERY - MariaDB sustained replica lag on db1147 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1147&var-port=9104 [23:11:01] PROBLEM - MariaDB sustained replica lag on db1143 is CRITICAL: 39.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1143&var-port=9104 [23:11:19] PROBLEM - MariaDB sustained replica lag on db1146 is CRITICAL: 117.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1146&var-port=13314 [23:14:55] RECOVERY - MariaDB sustained replica lag on db1146 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1146&var-port=13314 [23:18:13] RECOVERY - MariaDB sustained replica lag on db1143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1143&var-port=9104 [23:36:43] PROBLEM - MariaDB sustained replica lag on db1148 is CRITICAL: 83.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1148&var-port=9104 [23:37:13] PROBLEM - MariaDB sustained replica lag on db1121 is CRITICAL: 82.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [23:38:15] PROBLEM - MariaDB sustained replica lag on db1147 is CRITICAL: 41.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1147&var-port=9104 [23:39:29] RECOVERY - MariaDB sustained replica lag on db1148 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1148&var-port=9104 [23:39:39] PROBLEM - MariaDB sustained replica lag on db1141 is CRITICAL: 87.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1141&var-port=9104 [23:41:05] RECOVERY - MariaDB sustained replica lag on db1141 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1141&var-port=9104 [23:44:29] RECOVERY - MariaDB sustained replica lag on db1121 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1121&var-port=9104 [23:45:41] RECOVERY - MariaDB sustained replica lag on db1147 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1147&var-port=9104 [23:55:59] PROBLEM - MariaDB sustained replica lag on db1143 is CRITICAL: 50.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1143&var-port=9104