[00:59:11] 10DBA, 10MediaWiki-General-or-Unknown, 10Operations, 10MW-1.31-release-notes (WMF-deploy-2018-02-20 (1.31.0-wmf.22)), and 2 others: Regularly purge expired temporary userrights from DB tables - https://phabricator.wikimedia.org/T176754#4016257 (10EddieGP) Sorry, I was about to ask when exactly to do this o... [03:22:20] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T188187#4016491 (10Cmjohnson) I have one left and now I see db1064 is degraded. We needed to order more @robh. [06:34:55] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T188187#4016615 (10Marostegui) >>! In T188187#4016491, @Cmjohnson wrote: > I have one left and now I see db1064 is degraded. We needed to order more > @robh. Yeah...just saw that. Let's save that spare disk for db... [06:37:00] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1064 - https://phabricator.wikimedia.org/T188685#4016618 (10Marostegui) p:05Triage>03Normal This is a slave in s4. There is only one spare disk left and we will use it for db1068 (s4 master - T188187#4016615) so we need to order more as per @Cmjohnson... [06:42:22] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089#4016623 (10Marostegui) >>! In T187089#4015265, @Anomie wrote: > This seems to be moving along reasonably q... [06:56:25] 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Patch-For-Review, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#4016662 (10Marostegui) >>! In T184666#4016082, @Reedy wrote: > > One thing to check is that s7 has no immediate sto... [07:55:41] 10DBA, 10Wikimedia-Incident: Investigate why query killer didn't kill 1-hour long queries - https://phabricator.wikimedia.org/T188505#4016738 (10Jonas) Is there any progress so far? Is someone actively working on this? [07:57:01] 10DBA, 10Wikimedia-Incident: Investigate why query killer didn't kill 1-hour long queries - https://phabricator.wikimedia.org/T188505#4016739 (10Marostegui) >>! In T188505#4016738, @Jonas wrote: > Is there any progress so far? > Is someone actively working on this? No, we are not yet working on this. We have... [08:07:23] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, 10User-Daniel: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#4016752 (10Ladsgroup) [08:10:27] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, 10User-Daniel: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#4016756 (10Ladsgroup) Just noting that this is enabled eveyrwhere except commons, hywiki, cebwiki, and warwiki [08:31:58] 10DBA, 10MediaWiki-Watchlist, 10Wikidata, 10Performance, and 3 others: re-enable Wikidata Recent Changes integration on Russian Wikipedia - https://phabricator.wikimedia.org/T179012#3710122 (10Ladsgroup) This is done for several days now. I enabled it in [[https://gerrit.wikimedia.org/r/#/c/415078/ | gerri... [08:32:32] 10DBA, 10MediaWiki-Watchlist, 10Wikidata, 10Performance, and 4 others: re-enable Wikidata Recent Changes integration on Russian Wikipedia - https://phabricator.wikimedia.org/T179012#4016819 (10Ladsgroup) a:03Ladsgroup [10:38:06] I will wait until labsdb1010 is at >95% disk occupation to delete the existing files [10:38:30] that will give us 2TB of "good files" [10:39:32] if everthing is ok, the fact that 7TB on filesystem takes 1.8TB compressed is worrying [10:41:02] oh wow 7TB already in labs [10:41:25] we should check compression [10:41:44] Yeah, we should probably give it a go (or optimizes) [10:41:46] on all hosts, at some point [10:42:10] I don't think that s3 taking 2TB is normal [10:42:19] how's cebwiki behaving? [10:42:58] no idea [10:43:06] not sure if it is that, or wikidata tracking [10:44:13] https://grafana.wikimedia.org/dashboard/db/prometheus-cluster-breakdown?panelId=1087&fullscreen&orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=mysql&var-instance=All&from=1504349048000&to=1519987448001 [10:44:50] db1099? is that s3? [10:44:56] or what do you mean? :) [10:45:16] I think the share link doesn't work well [10:46:54] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#4017041 (10Marostegui) db1114 is now fully pooled in s1 and db1073 is depooled. Let's see how it goes during the weekend and then move db1073 to m5. [10:48:30] BTW, the test dbs were supposed to be one on each datacenter [10:49:09] so we actually had db1114 available, we just wanted to move one host from codfw to equiad virtually [10:49:24] ah that's true, good point [10:52:17] 10DBA, 10MediaWiki-Watchlist, 10Wikidata, 10Performance, and 4 others: re-enable Wikidata Recent Changes integration on Russian Wikipedia - https://phabricator.wikimedia.org/T179012#4017070 (10Elitre) Are you going to announce anything about this? [11:06:27] there seems to be a partition on enwikinews/pagelinks @ labs [11:09:09] in any case, the compression seems to be 5x, and 1.7*5 = 8.5, so deleting the original files [11:11:14] wow, 5x is pretty good [11:11:22] we need to check the optimizing/compressing [11:11:59] well, I see it the other way [11:12:09] 5x is bad for the original [11:12:24] well, yes [11:12:39] BTW, did you see that they closed the gtid ticket you sent? [11:13:24] Yeah, but that was long ago I think;: https://phabricator.wikimedia.org/T149418#3765720 [11:13:27] I commented there [11:13:47] Can't wait to test it XD [11:14:00] yeah, but we didn't have 10.1.30 back then [11:14:52] if we can enable them on multisource, even if it requires cleanups, it would be a huge win [11:15:08] (in case of labsdb crashes) [11:15:22] yeah, I wanted to test, but is scary to enable it, but we have to try [11:15:32] at least in our labs :) [11:15:34] it would be also interesting [11:15:52] to cleanup the garbage gtids [11:16:20] for production hosts [11:25:51] yeah, it is getting 2-3 lines of gtids [11:43:56] jynus & marostegui -- este renombrado lleva 3 días funcionando: https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/Glorious_Engine [11:45:10] Hauskatze: how many wikidata edits? [11:45:35] 30,286 [11:46:34] they ain't that much [11:46:55] I suspect he's become stuck for some reason [11:47:26] mind running mwscript showJobs.php --wiki=wikidatawiki LocalRenameJob and see if there's anything there? [11:47:36] for RenameJob [11:48:01] LocalRenameUserJob* [11:48:03] sorry [11:48:10] https://wikitech.wikimedia.org/wiki/Stuck_global_renames [11:48:15] I'll create a task nonetheless [11:49:12] can you please ask for that on -operations? [11:49:13] Hauskatze: I would prefer if someone else with dev grants can run it, in case something goes wrong [11:49:23] ok [11:49:24] that is not really db-related [11:49:25] yep [11:49:32] Hauskatze: If something goes wrong I wouldn't really know what to do :) [11:49:34] sorry, wrong channel indeed [11:49:39] :) [11:49:46] the script just shows things but yeah, I can understand [11:49:56] I'm creating a task and poking the relevant people [11:50:04] Thank you :) [11:50:21] Hauskatze: Remember last time we did soemthing similar, there were more issues and we had to ping lego to jump in and fix it [11:52:13] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, 10User-Daniel: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#4017250 (10Lydia_Pintscher) 05Open>03Resolved [11:54:19] T188721 filed [11:54:20] T188721: Global rename of Erik_Fastman to Glorious_Engine stuck "in progress" since 28th February on wikidatawiki - https://phabricator.wikimedia.org/T188721 [11:54:38] Thanks! [11:55:41] in case you want to comment after having "sniffed" with root@localhost on wikidatawiki :-) [11:55:52] delete * from * xD [11:56:02] or 'drop' better [11:56:04] XDDD [11:56:15] see how much time they take to notice [11:56:23] * Hauskatze back to work, enough unfunny jokes [11:57:12] thanks for your work! [12:16:08] 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Patch-For-Review, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#4017331 (10jcrespo) I don't mind if global_properties get filled in, but would that mean, the local ones would get cl... [12:34:13] 10DBA, 10Schema-change, 10Tracking: [DO NOT USE] Schema changes for Wikimedia wikis (tracking) [superseded by #Blocked-on-schema-change] - https://phabricator.wikimedia.org/T51188#4017393 (10Anomie) [12:40:48] 10DBA, 10Patch-For-Review: Run pt-table-checksum on s1 (enwiki) - https://phabricator.wikimedia.org/T162807#4017417 (10Marostegui) ores_classification table is done. Next (and last, pending another full recheck): user [14:13:55] 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Patch-For-Review, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#4017706 (10Reedy) >>! In T184666#4017331, @jcrespo wrote: > I don't mind if global_properties get filled in, but woul... [14:21:51] 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Patch-For-Review, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#4017721 (10jcrespo) > I guess there'd be scope to have job queue jobs that go through and removes local user preferen... [14:27:05] 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Patch-For-Review, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#4017727 (10jcrespo) To clarify, I don't need a cleanup solution now, but I would like to see a plan was thought to av... [14:41:32] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#4017759 (10jcrespo) a:05jcrespo>03Marostegui Reflecting latest work [14:42:24] 10DBA, 10Patch-For-Review: Finish the database backups generation script to create consistent logical backups in CODFW - https://phabricator.wikimedia.org/T184696#4017761 (10jcrespo) a:03jcrespo [16:45:48] 10DBA, 10Wikimedia-Incident: Investigate why query killer didn't kill 1-hour long queries - https://phabricator.wikimedia.org/T188505#4018321 (10jcrespo) There is a strange gap on any kind of killing activity between november and march: ``` 171966558 | 2017-11-20 06:29:35 | wmf_slave_wikiuser_sleep | ki... [16:53:18] 10DBA, 10Cloud-VPS, 10Data-Services, 10cloud-services-team (Kanban): Add page_props.pp_value index to Wiki Replicas - https://phabricator.wikimedia.org/T140609#4018341 (10bd808) a:03aborrero Index creation commands still need to be run on the sanitarium side. Assigning to @aborrero for follow up. [16:56:28] 10DBA, 10Cloud-VPS, 10Data-Services, 10cloud-services-team (Kanban): Add page_props.pp_value index to Wiki Replicas - https://phabricator.wikimedia.org/T140609#4018355 (10aborrero) I talked with @Marostegui about doing this next week. [16:58:21] 10DBA, 10Cloud-VPS, 10Data-Services, 10cloud-services-team (Kanban): Add page_props.pp_value index to Wiki Replicas - https://phabricator.wikimedia.org/T140609#4018359 (10Marostegui) >>! In T140609#4018355, @aborrero wrote: > I talked with @Marostegui about doing this next week. Let's chat about it next w... [17:16:15] ETA 1:23:52 [17:27:31] 10DBA, 10Patch-For-Review, 10Wikimedia-Incident: Investigate why query killer didn't kill 1-hour long queries - https://phabricator.wikimedia.org/T188505#4018494 (10jcrespo) a:03jcrespo Running `SELECT sleep(70);` as wikiuser to check it is at least in some way working. [17:39:38] 10DBA, 10Patch-For-Review, 10Wikimedia-Incident: Investigate why query killer didn't kill 1-hour long queries - https://phabricator.wikimedia.org/T188505#4018569 (10jcrespo) The same query was killed: ``` MariaDB [wikidatawiki]> SELECT /* Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup::selectRevisionInf... [17:41:33] 10DBA, 10Patch-For-Review, 10Wikimedia-Incident: Investigate why query killer didn't kill 1-hour long queries - https://phabricator.wikimedia.org/T188505#4018574 (10jcrespo) The thesis is that for some reason, either the query killer was disabled or crashed, or other situation, that made it not working on th... [18:32:49] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Consider dropping the "wb_items_per_site.wb_ips_site_page" index - https://phabricator.wikimedia.org/T179793#4018800 (10jcrespo) 05Resolved>03Open a:05Marostegui>03hoo Apparently, the repo was never updated: https://phabricator.wikimedia.o... [18:36:37] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Consider dropping the "wb_items_per_site.wb_ips_site_page" index - https://phabricator.wikimedia.org/T179793#4018806 (10jcrespo) Adding @Ladsgroup because he seems to be our link with wikidata- this should be a 1-line commit (I haven't checked)- r... [21:07:17] 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Patch-For-Review, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#4019169 (10MaxSem) The model is the following: * When the user makes a setting global, a row is inserted into `centra... [23:48:04] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Consider dropping the "wb_items_per_site.wb_ips_site_page" index - https://phabricator.wikimedia.org/T179793#4019835 (10Ladsgroup) Will do it ASAP.