[06:01:01] 10DBA, 10Data-Services, 10MediaWiki-extensions-Babel, 10Patch-For-Review, 10cloud-services-team (Kanban): Replicate babel db table on Labs - https://phabricator.wikimedia.org/T160713#3108521 (10Marostegui) Hi Andrew, labsdb1009 and labsdb1010 have puppet disabled because: ``` The last Puppet run was at... [06:13:52] 10DBA, 10Operations, 10Performance-Team, 10Traffic, 10Wikidata: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3414623 (10jcrespo) Probably related: T169884 [07:36:49] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s1 - https://phabricator.wikimedia.org/T166204#3414757 (10Marostegui) db1083 is done: ``` root@neodymium:/home/marostegui# for i in `cat s1_tables`; do echo $i; mysql --skip-ssl -hdb1083 enwik... [07:37:02] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s1 - https://phabricator.wikimedia.org/T166204#3414758 (10Marostegui) [07:41:09] 10DBA, 10Operations, 10Performance-Team, 10Traffic, 10Wikidata: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3414762 (10jcrespo) > I also wonder why some of those log warnings come from close() and others have the proper commitM... [08:13:22] 10DBA, 10Analytics: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3414818 (10Marostegui) >>! In T153033#3413793, @Nuria wrote: >> I can write to the typical research lists to ask this question if you like, so this task doesn't stall. But if this situation comes up again we shou... [08:20:10] 10DBA, 10Operations, 10Wikimedia-Site-requests, 10User-MarcoAurelio: Global rename of Idh0854 → Garam: supervision needed - https://phabricator.wikimedia.org/T167031#3414823 (10MarcoAurelio) 05stalled>03Open a:03MarcoAurelio [08:57:36] 10DBA, 10Wikimedia-Language-setup: Deletion of als Wiktionary, Wikibooks and Wikiquote? - https://phabricator.wikimedia.org/T169450#3398639 (10Marostegui) Deleting a database just like that isn't easy and can create issues lots of issues. Last time we tried it with ptwikimedia it was a bit of a pain: T126832 a... [08:59:36] 10DBA, 10Operations: Evaluate how hard would be to get aa(wikibooks|wiktionary) and howiki databases deleted - https://phabricator.wikimedia.org/T169928#3413499 (10Marostegui) Deleting it just from the database can create inconsistencies. I wouldn't feel too comfortable just issuing a drop database in producti... [09:00:19] 10DBA, 10Wikimedia-Language-setup: Deletion of als Wiktionary, Wikibooks and Wikiquote? - https://phabricator.wikimedia.org/T169450#3398639 (10jcrespo) Redirection happens at apache or varnish level, so it is totally independent from the database. Only thing relevant is to move those to deleted lists, if they... [09:31:38] 10DBA, 10Operations, 10Wikimedia-Site-requests, 10User-MarcoAurelio: Global rename of Idh0854 → Garam: supervision needed - https://phabricator.wikimedia.org/T167031#3414996 (10MarcoAurelio) 05Open>03Resolved Thanks to @marostegui for his help. [09:59:38] 10DBA, 10Operations, 10Scoring-platform-team, 10cloud-services-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3415085 (10jcrespo) Someone announced 60 seconds of downtime, which I do not think is reasonable- rebooting fully a server and all its services takes around 3... [10:26:03] 10DBA, 10Patch-For-Review: Setup dbstore2002 with 2 new mysql instances from production and enable GTID - https://phabricator.wikimedia.org/T169510#3415116 (10Marostegui) dbstore2002 is now running two instances. The general one with s1,s3, s4 and s5 and another instance on port 3312 with s2 (compressed) Thi... [14:18:20] 10DBA, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Puppetize Piwik's Database and set up periodical backups - https://phabricator.wikimedia.org/T164073#3415609 (10elukey) A lot of things changed from my last post, most of them due to the fact that now the apps are not sending any... [14:40:54] 10DBA, 10Commons, 10MediaWiki-Special-pages, 10Wikimedia-General-or-Unknown, and 3 others: Special:ShortPages does not load in Wikimedia Commons: "Read timeout is reached" - https://phabricator.wikimedia.org/T168010#3415666 (10Aklapper) Wondering if T169908 is another instance of this? [15:12:01] 10DBA, 10Operations, 10Scoring-platform-team, 10cloud-services-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3415739 (10Halfak) Announcements have been updated. Thanks for the note. Shall we always announce a 1 hour maintenance window for DB maintenance? [15:15:14] 10DBA, 10Operations, 10Scoring-platform-team, 10cloud-services-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3415746 (10jcrespo) It varies from maintanance to maintenance, depending on the work to be done. Some take more some take less- the "normally" was meant as "N... [15:25:48] 10DBA, 10Commons, 10MediaWiki-Special-pages, 10Wikimedia-General-or-Unknown, and 3 others: Special:ShortPages does not load in Wikimedia Commons: "Read timeout is reached" - https://phabricator.wikimedia.org/T168010#3415783 (10Anomie) >>! In T168010#3415666, @Aklapper wrote: > Wondering if T169908 is anoth... [15:38:26] 10DBA, 10Commons, 10MediaWiki-Special-pages, 10Wikimedia-General-or-Unknown, and 3 others: Special:ShortPages does not load in Wikimedia Commons: "Read timeout is reached" - https://phabricator.wikimedia.org/T168010#3415816 (10jcrespo) I do not know much about mediawiki responsabilities- but shouln't this... [15:39:44] 10DBA, 10Operations, 10Scoring-platform-team, 10cloud-services-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3415818 (10Halfak) [15:40:12] 10DBA, 10Data-Services, 10MediaWiki-extensions-Babel, 10Patch-For-Review, 10cloud-services-team (Kanban): Replicate babel db table on Labs - https://phabricator.wikimedia.org/T160713#3108521 (10chasemp) Thanks @Marostegui you are a scholar and a gentlemen. Is this task done now? @Base can you confirm? [15:40:52] 10DBA, 10Operations, 10Scoring-platform-team, 10cloud-services-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3369055 (10Halfak) [15:41:01] 10DBA, 10Operations, 10Scoring-platform-team, 10cloud-services-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3369055 (10Halfak) Gotcha. Next time, we should add these details to the task description and I'll pick them up from there when making announcement. :) In... [15:41:53] After realising that upgrading db1102 to 10.1 isn't that easy (as it runs sanitarium role) and that would need a bit of change to get db1102 to get it installed, I am logging off :p [15:45:33] ok, bye! [15:47:11] 10DBA, 10Cloud-Services, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3415841 (10Marostegui) Before starting any replication for s2 or s6 on labs servers, we need to either upgrade db1102 to 10.1 (which will require... [15:51:46] 10DBA, 10Data-Services, 10MediaWiki-extensions-Babel, 10Patch-For-Review, 10cloud-services-team (Kanban): Replicate babel db table on Labs - https://phabricator.wikimedia.org/T160713#3415846 (10Andrew) Thanks @Marostegui [17:25:34] 10DBA, 10Data-Services, 10Patch-For-Review: Expose ar_content_format and ar_content_model columns of archive table on Labs replicas - https://phabricator.wikimedia.org/T89741#3416085 (10Umherirrender) a:03Umherirrender [17:30:24] 10DBA, 10Commons, 10MediaWiki-Special-pages, 10Wikimedia-General-or-Unknown, and 3 others: Special:ShortPages does not load in Wikimedia Commons: "Read timeout is reached" - https://phabricator.wikimedia.org/T168010#3352925 (10Catrope) I'll review this once I'm free, in about half an hour. As for the respo... [17:54:41] 10DBA, 10Data-Services, 10MediaWiki-extensions-Babel, 10Patch-For-Review, 10cloud-services-team (Kanban): Replicate babel db table on Labs - https://phabricator.wikimedia.org/T160713#3108521 (10bd808) a:03Andrew [18:01:20] 10DBA, 10Data-Services, 10MediaWiki-extensions-Babel, 10Patch-For-Review, 10cloud-services-team (Kanban): Replicate babel db table on Labs - https://phabricator.wikimedia.org/T160713#3416178 (10bd808) 05Open>03Resolved Checked with first alphabetic wiki from each shard: ``` $ sql enwiki (u3518@enwik... [18:03:54] 10DBA, 10Data-Services, 10Patch-For-Review, 10User-bd808, and 2 others: Drop ukwikimedia from labsdb hosts (was: ukwikimedia still present on replicas dbs on labs hosts) - https://phabricator.wikimedia.org/T169488#3416186 (10bd808) 05Open>03Resolved The config change was synced out. `ukwikimedia` is no... [20:17:03] 10DBA, 10Cloud-Services, 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: ukwikimedia_p revision table is apparently private - https://phabricator.wikimedia.org/T170005#3416711 (10Matthewrbowker) [20:18:21] 10DBA, 10Cloud-Services, 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: ukwikimedia_p revision table is apparently private - https://phabricator.wikimedia.org/T170005#3416156 (10Legoktm) uk.wikimedia.org is no longer hosted by Wikimedia and was removed from the database lists. Where are you getting the... [20:21:11] 10DBA, 10Cloud-Services, 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: ukwikimedia_p revision table is apparently private - https://phabricator.wikimedia.org/T170005#3416721 (10MusikAnimal) >>! In T170005#3416718, @Legoktm wrote: > uk.wikimedia.org is no longer hosted by Wikimedia and was removed from... [20:22:40] 10DBA, 10Cloud-Services, 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: ukwikimedia_p revision table is apparently private - https://phabricator.wikimedia.org/T170005#3416725 (10Matthewrbowker) meta_p.wiki: https://quarry.wmflabs.org/query/20104 [20:22:45] 10DBA, 10Cloud-Services, 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: ukwikimedia_p revision table is apparently private - https://phabricator.wikimedia.org/T170005#3416726 (10Legoktm) OK, it just needs to be removed from meta_p. [20:22:56] 10DBA, 10Cloud-Services, 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: ukwikimedia_p needs to be removed from meta_p table - https://phabricator.wikimedia.org/T170005#3416727 (10Legoktm) [20:23:18] 10DBA, 10Cloud-Services, 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: ukwikimedia_p needs to be removed from meta_p table - https://phabricator.wikimedia.org/T170005#3416731 (10Marostegui) This is probably related: T169488 @bd808 might be able to help here [20:29:33] 10DBA, 10Cloud-Services, 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: ukwikimedia_p needs to be removed from meta_p table - https://phabricator.wikimedia.org/T170005#3416740 (10MusikAnimal) [22:42:16] 10DBA, 10Cloud-Services, 10Wikimedia-Site-requests: Reduce watchlist_count threshold - https://phabricator.wikimedia.org/T150548#3417152 (10Dispenser) ```lang=sql /* Most watched unstarted RfAs */ SELECT SUBSTRING(wl_title, 24, 50) AS RfA_subpage, Watchers FROM watchlist_count LEFT JOIN page ON wl_namespace=... [23:21:44] 10DBA, 10Analytics: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#2867045 (10Tbayer) >>! In T153033#3413282, @demon wrote: > A //month//?! I can't imagine there's //any// useful data to be gathered out of this. MoodBar was a complete and absolute failure. Looks like @nuria alr... [23:29:40] 10DBA, 10Analytics: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3417315 (10demon) I understand that not all things are failures. But whatever, moving on.