[08:33:22] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Wikidata, and 5 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3533003 (10jcrespo) @thcipriani @aaron I know something was done yesterday, (thank you!), may I ask... [08:41:52] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3533023 (10Marostegui) s7 is now replicating on dbstore2001 with gtid [08:41:58] 10DBA, 10Epic: Meta ticket: Migrate multi-source database hosts to multi-instance - https://phabricator.wikimedia.org/T159423#3533026 (10Marostegui) [08:45:46] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 3 others: wikibase-addUsagesForPage doesn't batch properly - https://phabricator.wikimedia.org/T172015#3533027 (10Ladsgroup) It immediately got stopped after deploying wmf.14: https://logstash.wikimedia.org/goto/0c342c6e95f46... [08:46:09] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 3 others: wikibase-addUsagesForPage doesn't batch properly - https://phabricator.wikimedia.org/T172015#3533028 (10Ladsgroup) 05Open>03Resolved p:05Triage>03Normal [08:48:41] 10DBA, 10Operations, 10ops-eqiad: RAID crashed on db1078 - https://phabricator.wikimedia.org/T173365#3533031 (10Marostegui) @Cmjohnson you think the disk will arrive today? I wouldn't want to leave this host depooled for the weekend :-( We can always repool it even we a degraded array but not ideal I guess [08:51:11] 10DBA, 10Patch-For-Review: Productionize 22 new codfw database servers - https://phabricator.wikimedia.org/T170662#3533033 (10Marostegui) [08:58:10] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Wikidata, and 5 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3533047 (10Ladsgroup) The fix is deployed and so I think this task should be closed, can you see an... [09:01:58] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Wikidata, and 5 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3533052 (10Ladsgroup) [09:12:01] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Wikidata, and 5 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3533065 (10Marostegui) >>! In T164173#3533047, @Ladsgroup wrote: > The fix is deployed and so I thi... [09:27:44] marostegui: okay, thanks. Just let me know if something happened [09:27:56] sure, we will comment on the ticket :) [09:33:21] marostegui: BTW, the project standard for mediawiki is to close tickets when merged [09:33:28] Ah [09:33:30] that is a WMF issue, however [09:33:31] I didn't know that [09:33:38] so I agree with you to leave it open [09:33:58] I am just giving context because most devels will close when code merged [09:34:13] Sure sure [09:34:14] which makes sense "code done" [09:34:16] I appreciate it :) [09:34:26] for us, it is different [09:34:35] we need to fix issues, not deploy code [09:38:12] I am going to repoint dbstore2x to db2033 [09:39:22] ah good! [09:40:17] as I will have to disable GTID, I will take the opportunity to make sure it is enabled on all shards of dbstore2x [09:40:27] (after the operation) [09:40:48] I disabled and enabled things several times, and I am not sure I left it in a proper state [09:41:46] btw. I'm picking up https://phabricator.wikimedia.org/T171460 again as the bug related to it got resolved [09:42:03] suer [09:42:06] sure [09:42:28] BTW, as it is a first run [09:42:41] maybe not the best idea to leave it running during weekend [09:42:49] on wikidatawiki, I mean [09:43:05] Yeah, that is also true, maybe let's start fresh on monday [09:43:09] I just do test runs today and finishing off the puppet patch [09:43:13] yes [09:43:15] that is ok [09:43:16] let's merge and deploy in Monday [09:43:19] exactly [09:43:29] test runs more than ok [09:43:37] even as long as you are connected [09:43:41] btw. It won't take 60 days as lots of slow parts got removed [09:44:09] but last time a cron job was left running it brought down a server because of a leak [09:44:14] and we weren't around [09:44:36] so better start a long running job on monday [09:44:46] yeah sure [09:44:52] so we have 5 days of monitoring stuff [09:54:25] 10DBA, 10Wikidata, 10Patch-For-Review: Populate term_full_entity_id on www.wikidata.org - https://phabricator.wikimedia.org/T171460#3533124 (10Ladsgroup) All properties have labels and: ``` ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermS... [10:40:04] I suppose I know restart instances of dbstore2001? [10:40:09] *now [10:40:36] sure [10:40:40] fine from my side [10:58:33] 10DBA, 10Wikimedia-Site-requests, 10Patch-For-Review, 10User-Urbanecm: Create English namespace aliases for hiwikiversity - https://phabricator.wikimedia.org/T172977#3533209 (10Jayprakash12345) [11:04:34] 10DBA, 10Wikimedia-Site-requests, 10Patch-For-Review, 10User-Urbanecm: Create English namespace aliases for hiwikiversity - https://phabricator.wikimedia.org/T172977#3514941 (10Marostegui) Hi, What are we, DBAs, supposed to be here @Jayprakash12345? [11:48:54] no garfana dashboard for s7, probably grants? [11:49:36] Ummm [11:49:37] I added them [11:49:42] let me recheck [11:49:51] yeah, I see the stats [11:50:00] maybe puppet didn't run on the prometheus hosts? [11:50:06] Ah could be [11:50:27] oh, I reloaded and now it is there [11:50:43] maybe it took a little bit to be deployed [11:50:52] yeah, i deployed at 12:30 [11:50:54] our time [12:50:13] 10DBA, 10Operations, 10ops-eqiad: Decommission db1015 - https://phabricator.wikimedia.org/T173570#3533347 (10Marostegui) [12:51:57] 10DBA, 10Operations, 10ops-eqiad: Decommission db1015 - https://phabricator.wikimedia.org/T173570#3533347 (10Marostegui) [13:10:11] Quick poll about the first runs of the eventlogging_cleaner.py [13:10:49] we did a couple of tests in https://phabricator.wikimedia.org/T156933#3531074 to measure the performance of UPDATE xyz WHERE uuid IN (list of uuids..) [13:11:39] and we'd like to test an alternative way, namely going back to the update in batches only using timestamps [13:12:27] we discarded it in the beginning since it was not guaranteeing batches with a precise number of elementts [13:12:45] but in light of the performance differences we might re-think it? [14:59:39] 10DBA, 10Wikidata, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Sprint: Populate term_full_entity_id on www.wikidata.org - https://phabricator.wikimedia.org/T171460#3533555 (10Ladsgroup) a:03Ladsgroup [16:35:19] 10DBA, 10Wikidata, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Sprint: Populate term_full_entity_id on www.wikidata.org - https://phabricator.wikimedia.org/T171460#3533769 (10Ladsgroup) Now it has been done up to Q200,000 ``` Processed up to page 191988 (Q194046) Processed up to page 192997 (Q195126... [18:13:35] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3534004 (10Bawolff) [22:05:47] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3534507 (10Bawolff)