[02:06:40] 10DBA, 10Wikidata: Optimize recentchanges and wbc_entity_usage table across wikis - https://phabricator.wikimedia.org/T187521#3977606 (10Ladsgroup) [06:39:53] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2048 - https://phabricator.wikimedia.org/T187419#3977674 (10Marostegui) 05Open>03Resolved All good now - thanks Papaul! ``` logicaldrive 1 (3.3 TB, RAID 1+0, OK) physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, OK) physicald... [06:42:01] 10DBA, 10Wikidata: Optimize recentchanges and wbc_entity_usage table across wikis - https://phabricator.wikimedia.org/T187521#3977606 (10Marostegui) p:05Triage>03Normal Thanks! Reducing disk size is always good news! :-) [07:20:33] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions, and 3 others: Schema change to prepare for dropping archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T185128#3977713 (10Marostegui) [07:20:57] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Perform schema change to add externallinks.el_index_60 to all wikis - https://phabricator.wikimedia.org/T153182#3977714 (10Marostegui) [07:33:30] 10DBA, 10Wikidata: Optimize recentchanges and wbc_entity_usage table across wikis - https://phabricator.wikimedia.org/T187521#3977723 (10GoranSMilovanovic) @Ladsgroup @Marostegui I have a cron job on **stat1004** that Sqoops the `wbc_entity_usage` tables for all projects into a HiveQL table for the [[ http:/... [07:34:50] 10DBA, 10Wikidata: Optimize recentchanges and wbc_entity_usage table across wikis - https://phabricator.wikimedia.org/T187521#3977724 (10Marostegui) >>! In T187521#3977723, @GoranSMilovanovic wrote: > @Ladsgroup @Marostegui I have a cron job on **stat1004** that Sqoops the `wbc_entity_usage` tables for all p... [07:39:49] 10DBA, 10Wikidata: Optimize recentchanges and wbc_entity_usage table across wikis - https://phabricator.wikimedia.org/T187521#3977606 (10jcrespo) > Sqoops the wbc_entity_usage tables Also, which db server is being used? analytics-replica/analytics store/dbstore1002 is ok to do that, others are not. [07:41:46] 10DBA, 10Wikidata: Optimize recentchanges and wbc_entity_usage table across wikis - https://phabricator.wikimedia.org/T187521#3977740 (10GoranSMilovanovic) @Marostegui m = 0, h = 0, dom = 7,14,21,29, mon = *, dow = *, i.e. every 7th, 14th, 21st, and 29th of the month, 00:00 UTC. [07:42:14] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions, and 3 others: Schema change to prepare for dropping archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T185128#3977743 (10Marostegui) s6 progress: [] dbstore2001 [] db2089 [] db2087 [] db2076 [] db20... [07:42:25] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Perform schema change to add externallinks.el_index_60 to all wikis - https://phabricator.wikimedia.org/T153182#3977744 (10Marostegui) s6 progress: [] dbstore2001 [] db2089 [] db2087 [] db2076 [] db2046 [] db2053 [] db2060 [] db2067... [07:42:41] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089#3977745 (10Marostegui) s6 progress: [] dbstore2001 [] db2089 [] db2087 [] db2076 [] db2046 [] db2053 []... [07:46:06] 10DBA, 10Wikidata: Optimize recentchanges and wbc_entity_usage table across wikis - https://phabricator.wikimedia.org/T187521#3977752 (10GoranSMilovanovic) @Marostegui The [[ https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/wmde/WDCM/+/master/WDCM_Sqoop_Clients.R | R script ]] that orchestrates Apache... [07:52:23] 10DBA, 10Epic: Meta ticket: Migrate multi-source database hosts to multi-instance - https://phabricator.wikimedia.org/T159423#3977758 (10elukey) >>! In T159423#3976734, @Marostegui wrote: > So just to be clear, you just want 3 hosts and also decommission dbstore1002 - meaning that there will be no redundancy f... [08:29:42] both db1053 (s2) and db2042 (s1) were found to be equal to its master [08:29:48] \o/ [08:32:23] 10DBA, 10Epic: Meta ticket: Migrate multi-source database hosts to multi-instance - https://phabricator.wikimedia.org/T159423#3977815 (10Marostegui) All clear! Thanks :-) Then 3 hosts should be it. [08:33:21] 10DBA, 10Epic: Meta ticket: Migrate multi-source database hosts to multi-instance - https://phabricator.wikimedia.org/T159423#3977816 (10jcrespo) Or 2 + eventlogging, more realistically, with the current budget. [09:45:27] 10DBA, 10Operations, 10ops-eqiad: Disk #5 (count starts at #0) of db1111 has corrupted sectors - https://phabricator.wikimedia.org/T187526#3977928 (10jcrespo) [09:48:33] db1077 and db1078 seems to be complaining, but at low rate, about connection rate [09:48:43] that's s3, no? [09:49:10] yes [09:49:46] maybe we should increase db1072 load? [09:50:10] we can try [09:50:15] or [09:50:17] even [09:50:23] db1075 itself [09:50:23] or wait a sec to see if they get gone [09:50:43] I checked, it is an ongoing isssue for quite some time [09:50:49] it is only the jobqueue [09:50:51] Ah, then maybe let's try with db1072 first [09:50:58] could be not a db issue [09:51:05] (I always prefer to leave the master alone if there is something else to try) [09:51:19] but a client issue beacuse so many wikis on s3 [09:52:16] the errors are not to high, tough [10:04:26] quick check at current reimage list: db1115|db2036|db2034|db2078|db2090|db2092 what to do keep,fix, cancel? _ [10:04:50] db1115 needs to stay and db2092 too [10:05:02] what are the others? [10:05:19] ah no db2092 no, we need to add db2093 [10:05:53] are you testing strech upgrading or what is its story? [10:06:09] db2093? [10:06:12] of the low ones [10:06:18] probably left overs [10:06:24] db2034 was probably reimaged to become x1 master [10:06:31] ah, ok [10:06:54] so I cam remove at least db2034 [10:07:10] db2036 too, maybe? [10:07:12] from my side. db2036, db2034 ,db2078 and db2090 can be gone (not touching them) [10:07:26] and if you are going to commit it, if you can change db2092 and place db2093 instead, that'd be good :) [10:07:42] I can do that, that is why I asked [10:07:47] I will add you as reviewer [10:07:59] thanks [10:08:02] please check that list specifically [10:08:06] 10DBA, 10Patch-For-Review: Run pt-table-checksum on s1 (enwiki) - https://phabricator.wikimedia.org/T162807#3977992 (10Marostegui) I have finished the oldimage table. Next: revision (which is in a pretty good state, so I don't expect it to take long) [10:10:59] db2093 does not have a dhcp entry [10:11:06] I will leave it as is [10:11:42] I will not add db2093 [10:11:53] because that is software raid [10:12:10] (confusing isn't it?) :-) [10:12:49] or I can add it to raid1 [10:15:26] sure, basically it needs to s/tendril2001/db2093 [10:15:29] I can do it if you like [10:15:42] I added one, but didn't remove the tendril one [10:16:12] I can do that later [10:16:13] no worries [10:17:58] I see no tendril there [10:18:23] ah, maybe it wasn't added yet [10:18:38] https://gerrit.wikimedia.org/r/#/c/411198/ [10:34:19] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 codfw machines - https://phabricator.wikimedia.org/T183470#3854215 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on sarin.codfw.wmnet for hosts: ``` ['db2042.codfw.wmnet'] ``` The log can... [10:34:30] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#3978016 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['db1053.eqiad.wmnet'] ``` The log... [10:53:49] 10DBA, 10Operations, 10netops, 10ops-codfw: switch port configuration for tendril2001 - https://phabricator.wikimedia.org/T186172#3978036 (10Marostegui) Please change this to db2093 as we have decided to rename that host from tendril2001 to db2093 (T186123#3975533) Thanks and sorry for che changes! [10:57:20] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#3978043 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1053.eqiad.wmnet'] ``` and were **ALL** successful. [10:59:16] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 codfw machines - https://phabricator.wikimedia.org/T183470#3978052 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2042.codfw.wmnet'] ``` and were **ALL** successful. [12:47:15] m3 looks saner: https://dbtree.wikimedia.org/ [12:48:30] good job! :) [13:01:54] I am talking with mark, for reading, we will increase the size of x1 [13:01:59] next year [13:02:04] rather than a full separate section [13:02:18] That makes sense yea [13:02:24] so maybe with the refresh this year + next year we can setup an x1 as large as the other sections [13:02:37] and that should be enough for them [13:02:42] That makes a lot more sense than having an extra section yeah - agreed [13:02:55] even specialize 1 replica just for them [13:02:59] if it was needed [13:03:16] 'readinglist' replica, if it was necessary [13:03:43] yeah, makes sense to me [13:03:46] otherwise it would be much more expensive and more overhead [13:04:02] yeah, and another set of servers to maintain [13:12:18] yep [13:12:20] good [13:12:27] i'm almost done with capex now [13:12:33] :-) [13:17:10] 10DBA, 10MediaWiki-General-or-Unknown, 10Operations, 10MW-1.31-release-notes (WMF-deploy-2018-02-20 (1.31.0-wmf.22)), and 2 others: Regularly purge expired temporary userrights from DB tables - https://phabricator.wikimedia.org/T176754#3978507 (10EddieGP) Next steps: # Deploy https://gerrit.wikimedia.org/r... [13:23:21] only 6-7 hosts left to finish our part of the decommissioning goal [13:35:27] :-) [13:54:25] cleaned up all ongoing issues, only db2033 warning to fix next week [16:30:13] 10DBA, 10Operations, 10netops, 10ops-codfw: switch port configuration for tendril2001 - https://phabricator.wikimedia.org/T186172#3979092 (10ayounsi) No worries, port description renamed! [17:37:26] 10DBA, 10Operations, 10hardware-requests, 10ops-codfw, 10Patch-For-Review: Decommission db2012 - https://phabricator.wikimedia.org/T187543#3979295 (10RobH) [17:37:41] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1043 - https://phabricator.wikimedia.org/T187542#3979298 (10RobH) [17:39:12] 10DBA, 10Operations, 10hardware-requests, 10ops-codfw, 10Patch-For-Review: Decommission db2012 - https://phabricator.wikimedia.org/T187543#3978461 (10RobH) Since this is pending the DBA team's work on stating the new host is online, I've appended in the #DBA flag. Once the DBA team work is done (their s... [17:39:18] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1043 - https://phabricator.wikimedia.org/T187542#3978426 (10RobH) Since this is pending the DBA team's work on stating the new host is online, I've appended in the #DBA flag. Once the DBA team work is done (their s... [19:00:08] 10DBA, 10MediaWiki-extensions-Linter, 10Patch-For-Review: Display count of remaining content space errors - https://phabricator.wikimedia.org/T173943#3979543 (10Legoktm) So, estimateRowCount doesn't appear like it will work: ``` wikiadmin@db1080(enwiki)>select count(*) from linter inner join page on page_id=... [20:02:59] 10DBA, 10MediaWiki-Debug-Logger: Create ip_logging table to query for logged actions by IP ranges - https://phabricator.wikimedia.org/T187579#3979688 (10MusikAnimal) [20:04:32] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging: Create ip_logging table to query for logged actions by IP ranges - https://phabricator.wikimedia.org/T187579#3979707 (10Legoktm) [20:05:43] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging: Create ip_logging table to query for logged actions by IP ranges - https://phabricator.wikimedia.org/T187579#3979688 (10Bawolff) Wouldn't this not work for the intended usecase of T146628 as that's asking for the target of the block log entry, not the user as... [20:14:06] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging: Create ip_logging table to query for logged actions by IP ranges - https://phabricator.wikimedia.org/T187579#3979739 (10MusikAnimal) >>! In T187579#3979710, @Bawolff wrote: > Wouldn't this not work for the intended usecase of T146628 as that's asking for the... [20:21:21] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging: Create ip_logging table to query for logged actions by IP ranges - https://phabricator.wikimedia.org/T187579#3979768 (10Bawolff) It would depend on how important having the ipc_timestamp in the index actually is (since log_search doesn't have that. OTOH if th... [21:51:07] 10DBA, 10MediaWiki-Database: Investigation: Using log_search to query for logged actions against IPs in a given range - https://phabricator.wikimedia.org/T187584#3979899 (10MusikAnimal) [21:58:22] 10DBA, 10MediaWiki-Database: Investigation: Using log_search to query for logged actions against IPs in a given range - https://phabricator.wikimedia.org/T187584#3979916 (10Bawolff) > On enwiki there are some 3.5 million blocks of IPs in logging -- a rough guess judging by SELECT COUNT(*) FROM logging WHERE lo... [22:00:31] 10DBA, 10MediaWiki-Database: Investigation: Using log_search to query for logged actions against IPs in a given range - https://phabricator.wikimedia.org/T187584#3979918 (10MusikAnimal) The other thing to note here is that at least for blocks within an IP range, we'll want any blocks of //subranges// to appear... [22:05:03] 10DBA, 10MediaWiki-Database: Investigation: Using log_search to query for logged actions against IPs in a given range - https://phabricator.wikimedia.org/T187584#3979941 (10MusikAnimal)