[00:27:10] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) [07:06:19] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10jcrespo) Please rename the proposal- we have absolutely no issue with this being a new namespace or how it is named... [07:17:16] 10DBA, 10Operations, 10Traffic, 10Patch-For-Review: Framework to transfer files over the LAN - https://phabricator.wikimedia.org/T156462 (10jcrespo) The original scope isn't met by far: * No throttling except it is easy to implement with pv * It is not intelligent * Compression is only on/off, not configur... [07:41:19] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4455344, @jcrespo wrote: > Please rename the proposal- we have absolutely no issue with this... [09:57:54] 10DBA, 10Wikidata: Provision a separate DB shard for wbc_entity_usage - https://phabricator.wikimedia.org/T176277 (10hoo) 05Open>03declined >>! In T176277#4449950, @Marostegui wrote: > @hoo is this still something to be done? No, I think we fine for now *crosses fingers* [10:01:12] 10DBA, 10monitoring, 10Wikimedia-Incident: Monitor read_only variable and/or uptime on database masters, make it page - https://phabricator.wikimedia.org/T172489 (10Marostegui) This plugin does exactly what we need for now: https://github.com/Napsty/check_mysql_readonly/blob/master/check_mysql_readonly.sh so... [10:27:46] 10DBA, 10monitoring, 10Wikimedia-Incident: Monitor read_only variable and/or uptime on database masters, make it page - https://phabricator.wikimedia.org/T172489 (10jcrespo) Note that doesn't have support for socket, mine above does (plus it adds some extra checks such us TLS, TLS expiration, etc.). [10:29:37] 10DBA, 10monitoring, 10Wikimedia-Incident: Monitor read_only variable and/or uptime on database masters, make it page - https://phabricator.wikimedia.org/T172489 (10jcrespo) Example: ``` root@neodymium:~$ ./check_mariadb.py -h db1067 --slave-status --primary-dc=eqiad --icinga --check_read_only=0 Version 10... [10:31:27] 10DBA, 10monitoring, 10Wikimedia-Incident: Monitor read_only variable and/or uptime on database masters, make it page - https://phabricator.wikimedia.org/T172489 (10Marostegui) >>! In T172489#4455705, @jcrespo wrote: > Note that doesn't have support for socket, mine above does (plus it adds some extra checks... [10:34:07] 10DBA, 10monitoring, 10Wikimedia-Incident: Monitor read_only variable and/or uptime on database masters, make it page - https://phabricator.wikimedia.org/T172489 (10jcrespo) > detects automatically whether the host is a slave or a master and returns OK or CRITICAL based on that That is actually another reas... [10:34:44] 10DBA, 10monitoring, 10Wikimedia-Incident: Monitor read_only variable and/or uptime on database masters, make it page - https://phabricator.wikimedia.org/T172489 (10Marostegui) That's true [10:37:13] 10DBA, 10monitoring, 10Wikimedia-Incident: Monitor read_only variable and/or uptime on database masters, make it page - https://phabricator.wikimedia.org/T172489 (10jcrespo) So, not against implementing this, but I would prefer if you could help me make check_mariadb.py work well, even if we need a rewrite (... [10:54:52] 10DBA, 10monitoring, 10Wikimedia-Incident: Monitor read_only variable and/or uptime on database masters, make it page - https://phabricator.wikimedia.org/T172489 (10Marostegui) I thought it would be a fast way of implementing this (as I still think it is important to monitor read_only flag and can bite us ag... [10:59:36] 10DBA, 10monitoring, 10Wikimedia-Incident: Monitor read_only variable and/or uptime on database masters, make it page - https://phabricator.wikimedia.org/T172489 (10jcrespo) I am actyally ok with doing this, it is your call, but it will not be quick because we still need etcd integration. I would be happy if... [10:59:50] 10DBA: Make sure multi-instance slaves page - https://phabricator.wikimedia.org/T200509 (10Marostegui) p:05Triage>03Normal [11:02:42] ^ not sure if we prepared the multi-instance slaves for paging or if it needs refactor, but I thought I would create a task so we don't forget about it [14:15:23] 10DBA, 10MediaWiki-API, 10Performance: list=logevents slow for users with last log action long time ago - https://phabricator.wikimedia.org/T71222 (10Marostegui) This is most likely due to: T71127 - this is pretty much fixed everywhere due to all the recent reimages, cloning, indexes fixing and all that. The... [14:39:06] 10DBA, 10MediaWiki-API, 10Performance: list=logevents slow for users with last log action long time ago - https://phabricator.wikimedia.org/T71222 (10Marostegui) This is yet another case of the optimizer doing different things: ``` root@neodymium:/home/marostegui# diff -u <(mysql.py -hdb1100 dewiki -e "show... [15:00:42] 10DBA, 10MediaWiki-API, 10Performance: list=logevents slow for users with last log action long time ago - https://phabricator.wikimedia.org/T71222 (10Marostegui) As expected nothing to do with versions, 10.1.34 behaves the same way as 10.1.33 - so something related to 10.: ``` root@neodymium:/home/marostegui... [15:41:34] 10DBA, 10MediaWiki-API, 10Performance: list=logevents slow for users with last log action long time ago - https://phabricator.wikimedia.org/T71222 (10Anomie) >>! In T71222#4456374, @Marostegui wrote: > db1100 is the only host in eqiad (apart from rc slaves) that has: > ``` > KEY `log_user_type_time` (`log_us... [15:47:47] 10DBA, 10MediaWiki-API, 10Performance: list=logevents slow for users with last log action long time ago - https://phabricator.wikimedia.org/T71222 (10Marostegui) >>! In T71222#4456488, @Anomie wrote: >>>! In T71222#4456374, @Marostegui wrote: >> db1100 is the only host in eqiad (apart from rc slaves) that ha... [15:56:50] 10DBA, 10MediaWiki-Database, 10MediaWiki-Special-pages, 10Wikimedia-log-errors: Special:Contributions: "contributions page filtered for namespace or RevisionDeleted edits" error - https://phabricator.wikimedia.org/T108782 (10Marostegui) 05Open>03Resolved I am going to consider this resolved. As Jaime s... [16:12:46] 10DBA, 10MediaWiki-History-or-Diffs, 10MediaWiki-Special-pages, 10Wikimedia-General-or-Unknown: Use a random timestamp on Special:NewFiles on Commons doesn't loads the page (504 Gateway Timeout) - https://phabricator.wikimedia.org/T163681 (10Marostegui) 05Open>03Resolved Since 24th April 2017 till now... [16:23:17] marostegui: BTW, it is my theory that schema changes, formal special replicas + 10.1 has fixed lots of perf issues [16:23:27] and only introduced a few ones [16:23:50] but we have been migrating for so long (+1 year plus) that we lost track [16:25:38] Yeah, that is why I was going thru some of the oldest stuff in the triage column [16:25:45] To see if the situation has changed [16:25:50] thanks [16:26:36] No, thank you for all the effort in the migration! [16:30:11] it will take a few hours for db1094 to be warmed up, I will deploy https://gerrit.wikimedia.org/r/448532 later [16:34:01] 10DBA, 10Cloud-Services, 10Cloud-VPS, 10Operations, 10Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359 (10jcrespo)