[14:06:50] I'm running the update-views script on wikireplicas (T371486) [14:06:51] T371486: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486 [14:07:34] I tried running it without any filter (good idea? bad idea? :P) and it failed with a permission error on this db: [14:07:37] Access denied for user 'maintainviews'@'localhost' to database 'gorwikiquote\\_p' [14:12:22] I'm re-running the cookbook with a db filter, but is that permissions error something that we should fix? [14:27:05] https://youtu.be/bsQwnboqZwQ [14:30:34] I don't think you may be able to do that until wikis are prepared T375095 [14:30:39] T375095: Post-creation work for gorwikiquote - https://phabricator.wikimedia.org/T375095 [14:31:20] you should skip new wikis until filters are in place- and specially NOT create any views on them until you have the ok from data persistence [14:31:43] as otherwise you run the risk of leaking private data [14:32:07] I guess it is https://phabricator.wikimedia.org/T375094 [14:32:26] that will happen next week, for now please skip all new wikis [14:33:18] ^ dhinus [14:33:29] jynus: ack [14:33:52] I opened T375760 and I'm testing a patch to filter by db [14:33:53] T375760: update-views cookbook doesn't handle filters correctly - https://phabricator.wikimedia.org/T375760 [14:34:49] another check that could be implemented (this is just a suggestion) [14:35:28] is to alert/skip/warn if the sanitization has not been done before adding the views [14:36:04] it is easy to check by seeing that there are no private fields and private tables [14:39:40] yeah that makes sense, I'll create a task [14:40:54] sadly because this is distubuted among 3 teams coordination issues are bound to happen, so better be poactive about possible missunderstandings 0:-) [14:41:02] *proactive [14:51:54] I did manage to update the view by filtering by database [14:52:07] is there an easy way to check the "globalblocks" table only exists in the "centralauth" db? [14:52:45] as in, metadata query? yes- information_schema.tables you can query with sql by table name [14:53:06] right [14:53:23] so you can do a select table_schema FROM tables WHERE table_name='X' [14:54:20] or if you are looking for views, I_S has all that data querable [14:56:31] the first one worked, thanks! [14:56:58] I ran it in an-redacteddb across all sections, and I found 2 occurrences: centralauth and labswiki [14:57:47] I'm not sure if labswiki is relevant for this change, but I'll re-run update-views on labswiki for consistency [14:58:27] labswiki shouldn't be there (yet) [14:58:44] I am not sure [14:59:24] nah, I am wrong it should be (on s6) [14:59:30] yes [14:59:35] but I don't think we should expose that table [14:59:41] until it it becomes SAL [14:59:51] *SUL [15:00:16] which I guess will be when that table gets dropped [15:00:38] give it a look to double check it is safe [15:00:48] it's exposed but empty https://quarry.wmcloud.org/query/86565 [15:00:53] cool [15:01:04] then no worries [15:02:13] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 10.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [15:02:25] let me comment it as something that may be a leftover [15:04:13] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 19.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [15:04:45] dhinus: what's your phab nick? [15:06:59] T161859#10180058 [15:07:00] T161859: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859 [15:07:30] I suscribed you by accident there, feel free to unsusribe/silence the task [15:11:13] RECOVERY - MariaDB sustained replica lag on s1 on db1206 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [15:14:16] jynus: sorry in a meeting [15:18:13] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 37.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [15:18:33] db1206 is dumps, I depooled but I may have to repool it again [15:21:13] RECOVERY - MariaDB sustained replica lag on s1 on db1206 is OK: (C)10 ge (W)5 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [15:33:19] So, what is is that needs to happen when check_private_data_report finds private data? [15:35:15] a script has to be run, and maybe sanitarium hosts restarted for new config, I think it is documented somewhere [15:35:26] Arn*ud said he would do it next week [15:36:27] https://wikitech.wikimedia.org/wiki/MariaDB/PII [15:48:56] bah my attempt of running update-views on labswiki for consistency ended in "Table 'labswiki.globaluser' doesn't exist" [15:54:23] I have no idea why that table is not there, or which labswiki views should exist in replicas [16:00:55] Sorry I cannot be of more help, I can help you with SQL or production questions, but we don't handle wikireplicas (since I think 7 years ago) [16:01:22] I think the views were setup by the previous cloud manager [16:02:46] jynus: that's cool [16:03:07] I went looking for docs and didn't find anything specific or actionable [16:03:39] and I would need something that was pretty specific and pretty actionable (a cookbook would be ideal) [16:03:40] urandom: as I said, Arnau*d already said he was going to handle those, see Wed email [16:03:55] jynus: thanks you've already been helpful :) I'm opening a couple tasks about the remaining mysteries! [16:04:00] urandom: "Re: Private data found at an-redacteddb1001" [16:04:12] I know, I was just hoping to be helpful and take a bit of pressure off him in his absence [16:04:22] (if possible) [16:04:46] sure, as long as you add it to me :-D [16:04:49] *don't [16:35:03] re: maintain-views issues, I created T375779 and T375780 [16:35:03] T375779: maintain-views: skip new databases that have not been sanitized yet - https://phabricator.wikimedia.org/T375779 [16:35:04] T375780: maintain-views fails on labswiki - https://phabricator.wikimedia.org/T375780 [19:54:15] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 72.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [19:58:15] RECOVERY - MariaDB sustained replica lag on s1 on db1206 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [20:12:15] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 54.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [20:20:48] FIRING: MysqlReplicationLagPtHeartbeat: MySQL instance db1206:9104 has too large replication lag (4m 24s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1206&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [20:21:48] FIRING: MysqlReplicationLag: MySQL instance db1206:9104@s1 has too large replication lag (4m 19s). Its replication source is db1163.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1206&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [20:25:48] FIRING: [2x] MysqlReplicationLagPtHeartbeat: MySQL instance db1206:9104 has too large replication lag (3m 42s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1206&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [20:35:48] RESOLVED: [2x] MysqlReplicationLagPtHeartbeat: MySQL instance db1206:9104 has too large replication lag (1m 57s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1206&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [20:36:48] RESOLVED: MysqlReplicationLag: MySQL instance db1206:9104@s1 has too large replication lag (1m 57s). Its replication source is db1163.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1206&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [20:38:15] RECOVERY - MariaDB sustained replica lag on s1 on db1206 is OK: (C)10 ge (W)5 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [23:28:15] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 31.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [23:32:15] RECOVERY - MariaDB sustained replica lag on s1 on db1206 is OK: (C)10 ge (W)5 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104