[06:16:55] 10DBA, 06Labs: S1 replag at 3 hours - https://phabricator.wikimedia.org/T163023#3185797 (10Marostegui) 05duplicate>03Resolved a:03jcrespo I believe this can be closed as all the slaves caught up after Jaime's fixes. ``` [root@labsdb1001 06:16 /root] # mysql --skip-ssl -e "show all slaves status\G" | egre... [06:36:10] 10DBA, 10Wikidata: Repeated reports of wikidatawiki (s5) API going read only - https://phabricator.wikimedia.org/T123867#1939840 (10Marostegui) >>! In T123867#3183910, @Multichill wrote: > In the last 10 minutes: > > pywikibot.data.api.APIError: readonly: The database has been automatically locked while the s... [07:07:21] 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3185828 (10Marostegui) dbstore1002 finished after 3 days of altering: ``` root@neodymium:~# mysql --skip-ssl -hdbstore1002 enwiki -e "show create table revision\G" ****... [07:17:08] 10DBA, 06Operations, 10hardware-requests, 10ops-codfw: codfw: (1) spare pool system for temp allocation as database failover - https://phabricator.wikimedia.org/T161712#3185829 (10Marostegui) I am moving this to "Done" on the DBA workboard as it is done from our side along with: T162290 so it is easier for... [07:56:25] 10DBA, 10MediaWiki-API, 06Operations, 10Traffic: Someone is parsing all enwiki pages using the action api at a rate of ~2M pages/hour - https://phabricator.wikimedia.org/T162129#3185844 (10Marostegui) 05stalled>03Resolved Looks like he hasn't come back again: https://grafana.wikimedia.org/dashboard/db/... [08:31:22] 10DBA, 07Epic, 13Patch-For-Review, 05codfw-rollout: Database maintenance scheduled while eqiad datacenter is non primary (after the DC switchover) - https://phabricator.wikimedia.org/T155099#3185865 (10Marostegui) I have been reviewing all the steps on https://wikitech.wikimedia.org/wiki/Switch_Datacenter... [09:04:01] 10DBA, 06Labs, 10Tool-Labs: labsdb1001 and labsdb1003 short on available space - https://phabricator.wikimedia.org/T132431#3185917 (10Marostegui) 37G of logfile in 4 days is quite a bit (we are logging warnings): ``` [root@labsdb1001 09:01 /srv/sqldata] # ls -lh labsdb1001.err -rw-r----- 1 mysql mysql 37G Ap... [09:56:30] 10DBA, 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Erwin's-tools, 10Tool-Labs-tools-Other: s51362 has been rate limited to 2 concurrent connections for creating hundreds of 1400-second queries to labsdb1001 and labsdb1003 every 10 seconds - https://phabricator.wikimedia.org/T162519#3185960 (10Akoopal) [09:58:22] 10DBA, 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Erwin's-tools, 10Tool-Labs-tools-Other: s51362 has been rate limited to 2 concurrent connections for creating hundreds of 1400-second queries to labsdb1001 and labsdb1003 every 10 seconds - https://phabricator.wikimedia.org/T162519#3165938 (10Akoopal) Did: t... [10:12:42] 10DBA, 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Erwin's-tools, 10Tool-Labs-tools-Other: s51362 has been rate limited to 2 concurrent connections for creating hundreds of 1400-second queries to labsdb1001 and labsdb1003 every 10 seconds - https://phabricator.wikimedia.org/T162519#3185981 (10Akoopal) From ir... [10:20:39] 10DBA, 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Erwin's-tools, 10Tool-Labs-tools-Other: s51362 has been rate limited to 2 concurrent connections for creating hundreds of 1400-second queries to labsdb1001 and labsdb1003 every 10 seconds - https://phabricator.wikimedia.org/T162519#3185989 (10valhallasw) The... [10:37:38] 10DBA, 06Operations: dbstore1001 troubleshoot IPMI issue - https://phabricator.wikimedia.org/T158893#3186029 (10Marostegui) Any news from Dell? Thanks! [13:08:18] 10DBA: Reclone db1063 to become a slave in s5 - https://phabricator.wikimedia.org/T163109#3186182 (10Marostegui) [13:30:23] 10DBA: Reclone db1068 to become a slave in s4 - https://phabricator.wikimedia.org/T163110#3186207 (10Marostegui) [13:51:11] 10DBA, 06Operations, 10Traffic: dbtree broken (for some users?) - https://phabricator.wikimedia.org/T162976#3182213 (10BBlack) tendril.wikimedia.org is independent of varnish, only dbtree.wikimedia.org (that we're talking about here) goes through the standard varnish stuff (although arguably tendril should b... [13:52:02] 10DBA, 06Operations, 10Traffic: dbtree broken (for some users?) - https://phabricator.wikimedia.org/T162976#3186237 (10BBlack) (also, generally speaking errors aren't cached, but in this case the error would be cached, because it's returned with a 200 status code...) [14:02:42] 10DBA, 06Revision-Scoring-As-A-Service, 10rsaas-articlequality : [Discuss] Hosting the monthly article quality dataset on labsDB - https://phabricator.wikimedia.org/T146718#3186263 (10Marostegui) Hi @Halfak What are the requirements then? The original 55GB initial dataset (split in different tables) or is... [14:21:03] 10DBA: Reclone db1063 to become a slave in s5 - https://phabricator.wikimedia.org/T163109#3186279 (10Marostegui) We need to check this: T161294#3174207 [14:22:42] 10DBA: Reclone db1068 to become a slave in s4 - https://phabricator.wikimedia.org/T163110#3186283 (10Marostegui) As per T162593#3175313 db1081 looks healthy to be used to clone db1068 and it would have the same data as the current master. [15:15:48] 10DBA, 06Revision-Scoring-As-A-Service, 10rsaas-articlequality : [Discuss] Hosting the monthly article quality dataset on labsDB - https://phabricator.wikimedia.org/T146718#3186337 (10Halfak) Hi @Marostegui. * Requirements are 15.6GB with an additional 2GB per year. See T146718#2677286 for the analysis *... [16:41:12] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 15User-Daniel: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3186644 (10Lydia_Pintscher) [18:32:22] 10DBA, 10fundraising-tech-ops: fundraising database tuning - https://phabricator.wikimedia.org/T158446#3187154 (10Jgreen) 05Open>03Resolved done [19:20:53] 10DBA, 06Operations, 10Traffic, 13Patch-For-Review: dbtree broken (for some users?) - https://phabricator.wikimedia.org/T162976#3187340 (10Dzahn) seems like we have 2 follow-ups: - make dbtree not use status code 200 for an error page - make wasat a working dbtree backend, then add it back to varnish dire... [19:22:25] 10DBA, 06Operations, 10Traffic, 13Patch-For-Review: dbtree broken (for some users?) - https://phabricator.wikimedia.org/T162976#3187342 (10Dzahn) @kaldari @bd808 Does dbtree work for you again? I am wondering if this ticket can be called resolved (if we follow-up with the things above i suppose). [19:45:14] 10DBA, 06Operations, 10Traffic, 13Patch-For-Review: dbtree broken (for some users?) - https://phabricator.wikimedia.org/T162976#3187405 (10bd808) >>! In T162976#3187342, @Dzahn wrote: > @kaldari @bd808 Does dbtree work for you again? I am wondering if this ticket can be called resolved (if we follow-up wit... [19:48:31] 10DBA, 06Operations, 10Traffic, 13Patch-For-Review: dbtree broken (for some users?) - https://phabricator.wikimedia.org/T162976#3187423 (10BBlack) Yeah the patch I deployed above should have fixed the issue in this ticket. Both of the suggested followups would be ideal, but probably aren't pressing at thi... [19:50:39] 10DBA, 06Operations, 10Traffic, 13Patch-For-Review: dbtree broken (for some users?) - https://phabricator.wikimedia.org/T162976#3187431 (10Dzahn) 05Open>03Resolved a:03Dzahn [19:50:56] 10DBA, 06Operations, 10Traffic, 13Patch-For-Review: dbtree broken (for some users?) - https://phabricator.wikimedia.org/T162976#3182213 (10Dzahn) a:05Dzahn>03None [19:52:15] 10DBA, 06Operations, 10Traffic, 13Patch-For-Review: dbtree broken (for some users?) - https://phabricator.wikimedia.org/T162976#3182213 (10Dzahn) a:03BBlack [19:57:06] 10DBA, 06Operations, 10Traffic: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#3187493 (10Dzahn) [20:01:19] 10DBA, 06Operations: dbtree: don't return 200 on error pages - https://phabricator.wikimedia.org/T163143#3187529 (10Dzahn) [21:56:22] 10DBA, 06Revision-Scoring-As-A-Service, 10rsaas-articlequality : [Discuss] Hosting the monthly article quality dataset on labsDB - https://phabricator.wikimedia.org/T146718#2669300 (10Seeeko) Thanks for the CC @Halfak, I'm watching this from my active volunteer account now too. We're hoping to use this for a...