[02:35:27] 10DBA, 07Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2500916 (10Phabricator_maintenance) [02:35:35] 10DBA, 06Labs, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2500925 (10Phabricator_maintenance) [05:17:39] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2501120 (10jcrespo) From TTO: > Well, it seems to work on testwiki. > > As I see it, there are no technical blockers to deploying this on other wi... [05:41:49] 10DBA, 10MediaWiki-extensions-ContentTranslation, 07WorkType-Maintenance: repeated errors with Content Translation: "A database error has occurred. Did you forget to run maintenance/update.php after upgrading?" - https://phabricator.wikimedia.org/T141501#2501122 (10Amire80) [05:42:57] 10DBA, 10MediaWiki-extensions-ContentTranslation, 07WorkType-Maintenance: repeated errors with Content Translation: "A database error has occurred. Did you forget to run maintenance/update.php after upgrading?" - https://phabricator.wikimedia.org/T141501#2501122 (10Amire80) p:05Triage>03High High priorit... [06:38:02] 10DBA, 10MediaWiki-extensions-ContentTranslation, 07Performance, 07WorkType-Maintenance: Investigate increase in "readonly" saving errors in Content Translation - https://phabricator.wikimedia.org/T141090#2501164 (10Nikerabbit) This error most likely comes from `ApiMain::checkReadOnly`, which calls `wfRead... [07:01:28] 10DBA, 10MediaWiki-extensions-ContentTranslation, 07Performance, 07WorkType-Maintenance: Investigate increase in "readonly" saving errors in Content Translation - https://phabricator.wikimedia.org/T141090#2501172 (10jcrespo) For fawiki, for example, [[ https://tendril.wikimedia.org/report/slow_queries?host... [07:03:07] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2501187 (10aaron) The more I think about it, the less I tend to like this. Is it possible to use a parser function that sets a page_props row to st... [07:21:07] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2501203 (10jcrespo) To be more constructive, instead of confrontational (as we all probably want the feature, the schema change is only a mean to an... [07:21:10] 10DBA, 10MediaWiki-extensions-ContentTranslation, 07Performance, 07WorkType-Maintenance: Investigate increase in "readonly" saving errors in Content Translation - https://phabricator.wikimedia.org/T141090#2501204 (10aaron) updateLinksTimestamp() just updates one row by PK. I might have lock contention on t... [07:26:24] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2501208 (10jcrespo) Note: I will *not*, of course, mark this as declined if there is active discussion, although I would recommend all discussions t... [07:26:40] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2501209 (10aaron) WikitextContent is just one Content subtype, if a different one is used, then that one can have it's own way to put the language i... [07:45:34] 10DBA, 10MediaWiki-extensions-ContentTranslation, 07Performance, 07WorkType-Maintenance: Investigate increase in "readonly" saving errors in Content Translation - https://phabricator.wikimedia.org/T141090#2501219 (10jcrespo) I should have clarified those are the the times I measured on the master, but they... [07:48:45] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2501237 (10jcrespo) I like that. [08:33:24] 10DBA, 10MediaWiki-extensions-ContentTranslation, 07Performance, 07WorkType-Maintenance: Investigate increase in "readonly" saving errors in Content Translation - https://phabricator.wikimedia.org/T141090#2501258 (10Nikerabbit) Was `updateLinksTimestamps` changed around 2016-07-12 when these started happen... [09:34:59] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2501293 (10Nikerabbit) This feature has shipped with MediaWiki since 1.24, albeit disabled by default and it had bugs that were fixed since 1.27 and... [10:41:07] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2501346 (10jcrespo) @Nikerabbit please have all the developers I mentioned before agreed to perform the changes and I will schedule it for the next... [11:45:44] 10DBA, 10MediaWiki-extensions-ContentTranslation, 03Language-Q1-2016-17 Sprint 3, 07Performance, 07WorkType-Maintenance: Investigate increase in "readonly" saving errors in Content Translation - https://phabricator.wikimedia.org/T141090#2501454 (10Amire80) [11:48:51] 07Blocked-on-schema-change, 10DBA, 06Operations, 06Revision-Scoring-As-A-Service, and 3 others: Remove oresc_rev index - https://phabricator.wikimedia.org/T140803#2501457 (10jcrespo) 05Open>03Resolved a:03jcrespo The schema change seems to have been succesful: ``` $ mysql -h s5-master wikidatawiki -e... [12:31:53] 10DBA, 10MediaWiki-Database, 07User-notice, 07Wikimedia-log-errors: Special:Block request makes the database timeout on db1055 - https://phabricator.wikimedia.org/T140650#2501665 (10jcrespo) After upgrade + engine_dependent_statistics fix + ANALYZE table, this issue is solved. It prefers `page_time`, but t... [12:53:34] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2501689 (10Nikerabbit) Ok I will see what I can do. [13:30:22] 10DBA, 10Continuous-Integration-Config, 10Datasets-General-or-Unknown, 06Release-Engineering-Team, and 3 others: Automatize the check and fix of object, schema and data drifts between production masters and slaves - https://phabricator.wikimedia.org/T104459#2501749 (10jcrespo) @ArielGlenn, before you devel... [14:50:57] 10DBA, 10MediaWiki-Database, 07User-notice, 07Wikimedia-log-errors: Special:Block request makes the database timeout on db1055 - https://phabricator.wikimedia.org/T140650#2501944 (10jcrespo) 05Open>03Resolved I do not see any more errors on blocking users since 2016-07-27T12:05:33 UTC. I believe this t... [15:18:48] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-API, 05MW-1.28-release-notes, and 2 others: ApiQueryRecentChanges::run is spiking, nuking API servers - https://phabricator.wikimedia.org/T140108#2502024 (10jcrespo) I am deploying this now to all WMF servers. [16:20:29] 10DBA, 10Flow, 03Collab-Team-2016-Apr-Jun-Q4, 03Collaboration-Team-Archive-2015-2016, 13Patch-For-Review: Cleanup ptwikibooks conversion - https://phabricator.wikimedia.org/T119509#2502188 (10jcrespo) Sorry for the delay, I am ready for the backups. The best way is to schedule some time- the backup proce... [16:29:39] 10DBA, 06Operations, 13Patch-For-Review: upgrade dbproxy1001/1002 to jessie - https://phabricator.wikimedia.org/T125027#2502223 (10jcrespo) @akosiaris I think now should be a safe time to restart bacula, but I will block on you giving an ok for that/confirming there are no ongoing jobs running. [16:33:59] 10DBA, 06Operations, 13Patch-For-Review: upgrade dbproxy1001/1002 to jessie - https://phabricator.wikimedia.org/T125027#2502243 (10akosiaris) It is. Let's do it. I already did a restart, everything is fine. [16:35:06] 10DBA, 06Operations, 13Patch-For-Review: upgrade dbproxy1001/1002 to jessie - https://phabricator.wikimedia.org/T125027#2502247 (10jcrespo) Oh, thank you a lot! I see bacula already from the new ip! Thank you again! [17:13:55] 10DBA, 06Labs, 06Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2502450 (10jcrespo) 05Open>03Resolved labsdb1002 itself has been decomissioned; the tracking of the new server setup (and labs fix in general) will be done at T140452 [17:30:56] 10DBA, 06Operations, 13Patch-For-Review: upgrade dbproxy1001/1002 to jessie - https://phabricator.wikimedia.org/T125027#2502549 (10jcrespo) 05Open>03Resolved Both servers are now in jessie. The only remaining trusty servers is dbproxy1002 and dbproxy1004, which will be upgraded to jessie when we perform... [17:35:18] 10DBA, 06Operations, 10ops-eqiad, 13Patch-For-Review: dbproxy1002 down - https://phabricator.wikimedia.org/T140983#2502581 (10jcrespo) 05Open>03Resolved No more issues from dbproxy1002, and we now have proxy redundancy. [17:39:21] 10DBA, 06Operations, 07Availability: Setup automatic failover for misc database servers - https://phabricator.wikimedia.org/T141547#2502618 (10jcrespo) [17:39:35] 10DBA, 06Operations, 07Availability: Setup automatic failover for misc database servers - https://phabricator.wikimedia.org/T141547#2502636 (10jcrespo) [17:39:38] 10DBA, 06Operations, 07Epic: Eliminate SPOF at the main database infrastructure - https://phabricator.wikimedia.org/T119626#1831329 (10jcrespo) [18:27:30] 10DBA, 06Labs, 10Labs-Infrastructure: Users can't run EXPLAIN queries to check the theoretical efficiency of their SQL - https://phabricator.wikimedia.org/T141095#2502814 (10zhuyifei1999) [18:27:45] 10DBA, 06Labs, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2502820 (10zhuyifei1999) [18:33:40] 10DBA, 06Labs, 10Labs-Infrastructure: Users can't run EXPLAIN queries to check the theoretical efficiency of their SQL - https://phabricator.wikimedia.org/T141095#2502834 (10zhuyifei1999) [18:33:43] 10DBA, 06Labs, 10Labs-Infrastructure, 07Epic, 07Tracking: Labs databases rearchitecture (tracking) - https://phabricator.wikimedia.org/T140788#2502833 (10zhuyifei1999) [20:25:17] 10DBA, 10Analytics, 13Patch-For-Review: Set up auto-purging after 90 days {tick} - https://phabricator.wikimedia.org/T108850#2503145 (10mforns) @Neil_P._Quinn_WMF @Nuria Thanks Neil for spotting this. I updated the docs on data retention and auto-purging. > One question: the main event logging page says tha... [20:42:23] 10DBA, 10Flow, 03Collab-Team-2016-Apr-Jun-Q4, 03Collaboration-Team-Archive-2015-2016, 13Patch-For-Review: Cleanup ptwikibooks conversion - https://phabricator.wikimedia.org/T119509#2503288 (10Mattflaschen-WMF) @matthiasmullie Are you planning to finish this up on the Flow side, or do you want to pass it... [22:17:54] 10DBA, 10Incident-20150205-SiteOutage, 06Operations, 05Wikimedia-Incident: sleeper database connection surges during outage - https://phabricator.wikimedia.org/T88770#2503670 (10greg)