[05:07:25] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment (The Letter Song), 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09): ipb_address_unique has an extra column in production but not in the code - https://phabricator.wikimedia.org/T251188 (10Marostegui) Thanks @Ladsgroup, however I am not seeing the difference... [06:07:42] 10DBA, 10Patch-For-Review: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy1018.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202008060607_maro... [06:43:08] 10DBA, 10Patch-For-Review: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1018.eqiad.wmnet'] ` and were **ALL** successful. [06:47:30] 10DBA, 10Patch-For-Review: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10Marostegui) [06:48:13] 10DBA, 10Product-Infrastructure-Team-Backlog, 10Release-Engineering-Team-TODO: Drop DB tables for now-deleted zerowiki from production - https://phabricator.wikimedia.org/T227717 (10Marostegui) a:03Marostegui [06:56:47] PROBLEM - 5-minute average replication lag is over 2s on db2098 is CRITICAL: 4739 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2098&var-port=13313&var-dc=codfw+prometheus/ops [07:02:15] RECOVERY - 5-minute average replication lag is over 2s on db2098 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2098&var-port=13313&var-dc=codfw+prometheus/ops [07:25:08] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 2020-09-14) rack/setup/install dbprov2003.codfw.wmnet - https://phabricator.wikimedia.org/T258749 (10jcrespo) Thank you Papaul very much for you work, really appreciated how fast this was completed! [08:13:55] 10DBA, 10Cloud-Services, 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09), 10Platform Team Initiatives (MCR Schema Migration), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) [08:33:06] 10DBA, 10Product-Infrastructure-Team-Backlog, 10Release-Engineering-Team-TODO: Drop DB tables for now-deleted zerowiki from production - https://phabricator.wikimedia.org/T227717 (10Marostegui) 05Open→03Resolved All tables truncated: ` abuse_filter +----------+ | count(*) | +----------+ | 0 | +---... [08:33:14] 10DBA, 10Epic, 10Tracking-Neverending: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) [08:50:10] I am going to run a logical backup into dbprov2003, then reimage it to test the new recipe that doesn't format everything for dbstores [08:52:00] jynus: cool :) [09:01:18] https://github.com/openark/orchestrator/issues/1227 :( [09:44:22] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment (The Letter Song), 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09): ipb_address_unique has an extra column in production but not in the code - https://phabricator.wikimedia.org/T251188 (10Ladsgroup) I have a feeling it's not commonswiki but it's testcommonswiki [09:50:50] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment (The Letter Song), 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09): ipb_address_unique has an extra column in production but not in the code - https://phabricator.wikimedia.org/T251188 (10Ladsgroup) Yup: ` wikiadmin@10.64.48.232(testcommonswiki)> show index... [10:01:02] hello, I've a question for you, dbmonitor[12]001 DNS records are set with TTL of 600s instead of the usual 1H. Is that required/on purpose? The tendril CNAME has 5M and that's totally normal [10:32:48] do you know who set it up like that? [10:34:19] blame says you, but may be a false detection [10:36:02] it used to be 1h when it was first setup [10:36:24] back in 2016 [10:37:23] actually no, it was copied from einsteinium, which I think it used to be the old icinga host [10:37:33] so maybe it was always like that [10:38:09] but that is the CNAME [10:39:00] no idea then [10:41:12] maybe it was done like that because ganety ips could change more frequently? no idea [10:44:31] kormat: reimage of dbprov woked nicely, in fact even the second disk sdb was left untouched and kept the mount point [10:45:30] great :) [10:53:48] jynus: it's one of 2 exceptions that we have, all the others are 1H, so if it's not intended I'd like to send a patch to make it 1H. The important one you want short I guess is the CNAME so that can be migrated at will [10:54:00] the VMs IPs should never change AFAIK [10:55:13] volans: nothing agains, but please CC manuel on patch, maybe he remember why [10:55:23] *remembers [10:55:43] sure, thx [12:11:37] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment (The Letter Song), 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09): ipb_address_unique has an extra column in production but not in the code - https://phabricator.wikimedia.org/T251188 (10Marostegui) codfw testcommonswiki done ` CREATE TABLE `ipblocks` ( `... [13:18:52] jynus kormat any objection if I take friday 14th off? [13:19:14] i have an objection if you _don't_ [13:19:18] XD [13:26:16] jynus, marostegui FYI the dbmonitor I mentioned earlier: https://gerrit.wikimedia.org/r/c/operations/dns/+/618744 [13:26:19] no hurry [13:32:58] ok [14:13:07] thanks a lot both! :) [14:13:26] one less discrepancy to keep track between automation and real life :) [14:22:23] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment (The Letter Song), 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09): ipb_address_unique has an extra column in production but not in the code - https://phabricator.wikimedia.org/T251188 (10Marostegui) s4 eqiad for testcommonswiki [] labsdb1012 [] labsdb1011... [16:06:02] 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment (The Letter Song), 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09): ipb_address_unique has an extra column in production but not in the code - https://phabricator.wikimedia.org/T251188 (10Ladsgroup) Thanks! The wiki is tiny so I hope you don't need to depool... [19:36:14] 10Blocked-on-schema-change: Schema change to make change_tag.ct_rc_id unsigned - https://phabricator.wikimedia.org/T259831 (10Ladsgroup) [19:39:33] 10Blocked-on-schema-change: Schema change to make change_tag.ct_rc_id unsigned - https://phabricator.wikimedia.org/T259831 (10Ladsgroup)