[05:27:41] federico3: For schema changes, can you put the progress on the task description similar to: https://phabricator.wikimedia.org/T399249 [07:09:27] marostegui: ok I'm updating it and the script above [07:10:30] federico3: thanks, it is useful to have the progress visible right away instead of having to dig thru all the comments [08:16:13] marostegui: is the naming convention fine with you? https://phabricator.wikimedia.org/T390087 [08:50:09] marostegui: can I start auto schema in s3? [09:43:09] I need a quick restart [11:20:55] federico3: yes [13:21:27] PROBLEM - MariaDB sustained replica lag on s7 on db2220 is CRITICAL: 50.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2220&var-port=9104 [13:22:07] PROBLEM - MariaDB sustained replica lag on s7 on db2208 is CRITICAL: 35.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2208&var-port=9104 [13:22:15] PROBLEM - MariaDB sustained replica lag on s7 on db2221 is CRITICAL: 28.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2221&var-port=9104 [13:23:43] PROBLEM - MariaDB sustained replica lag on s7 on db2168 is CRITICAL: 54 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2168&var-port=9104 [13:24:13] RECOVERY - MariaDB sustained replica lag on s7 on db2221 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2221&var-port=9104 [13:25:43] RECOVERY - MariaDB sustained replica lag on s7 on db2168 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2168&var-port=9104 [13:26:07] RECOVERY - MariaDB sustained replica lag on s7 on db2208 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2208&var-port=9104 [13:26:27] RECOVERY - MariaDB sustained replica lag on s7 on db2220 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2220&var-port=9104 [15:56:35] marostegui: the section_instances table associates one section to each instance.. is there a reason why this is a dedicated table instead of a column in the instances table? [15:58:32] federico3: can you guess? [15:58:38] did you count well? [16:00:11] an instance can replicate from multiple servers (multisource) [16:00:14] jynus: I ran SELECT count(*),instance FROM section_instances group by instance - there aren't instances with more than one section [16:00:22] is an instance or a server? [16:01:35] multisource- a single instance replicating from different other instances [16:02:06] https://mariadb.com/docs/server/ha-and-performance/standard-replication/multi-source-replication [16:02:16] (besides there few entries in the two tables missing a matching entry) [16:02:41] according to the groupby we are not using it (or maybe the table is out of date?) [16:03:13] anyhow as long as we know that we want to support it now or in future that answers my question :) [16:12:44] any strong feeling if I add 2 columns to the instances table for clarity and convenience? https://phabricator.wikimedia.org/P79853 [16:16:06] talk to amir, I wanted for a long time improve some things, but he was worried it could break some scripts [16:30:18] thanks [16:32:47] JennH: I am going to prepare es2036 and will ping you here (and on the task for the record) [16:33:18] ok ty! [16:40:25] JennH: es2036 ready and homer was run too [17:22:23] marostegui: finished it up. all yours! [17:26:21] JennH: all good, thanks [17:26:25] JennH: es2037 tomorrow? [17:33:13] sounds good [18:12:22] JennH: I'll have it ready for tomorrow [18:12:45] ty, i'll take care of it first thing [18:14:06] PROBLEM - MariaDB sustained replica lag on s7 on db2222 is CRITICAL: 29.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2222&var-port=9104 [18:14:16] PROBLEM - MariaDB sustained replica lag on s7 on db2221 is CRITICAL: 17.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2221&var-port=9104 [18:15:16] RECOVERY - MariaDB sustained replica lag on s7 on db2221 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2221&var-port=9104 [18:16:06] RECOVERY - MariaDB sustained replica lag on s7 on db2222 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2222&var-port=9104