[04:55:52] 10DBA, 10Data-Services, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3686331 (10bd808) [04:59:09] 10DBA, 10Data-Services: ores_classification table corrupt on enwiki labs replica labsdb1001 - https://phabricator.wikimedia.org/T178135#3686334 (10bd808) Adding the #dba tag, but the best answer is probably to use the new replicas instead of labsdb1001/1003 ({T142807}). [05:31:22] 10DBA, 10Data-Services, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3686350 (10Marostegui) [05:31:24] 10DBA, 10Data-Services: ores_classification table corrupt on enwiki labs replica labsdb1001 - https://phabricator.wikimedia.org/T178135#3686347 (10Marostegui) 05Open>03Resolved a:03Marostegui Hi, I have fixed this and also migrated the table to InnoDB which will hopefully prevent this corruption from h... [05:33:29] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3686351 (10Marostegui) [06:15:03] 10DBA, 10Patch-For-Review: Productionize 22 new codfw database servers - https://phabricator.wikimedia.org/T170662#3686393 (10Marostegui) [06:24:29] 10DBA, 10Data-Services: ores_classification table corrupt on enwiki labs replica labsdb1001 - https://phabricator.wikimedia.org/T178135#3686398 (10Bawolff) > But yes, the best answer is to start using the new servers :-) /me as a cloud services newb usually just uses whatever server that `$ sql enwiki_p` gets... [07:53:03] 10DBA, 10Patch-For-Review: Productionize 22 new codfw database servers - https://phabricator.wikimedia.org/T170662#3686494 (10Marostegui) [08:54:42] 10DBA, 10Patch-For-Review: Decommission db1050 - https://phabricator.wikimedia.org/T178162#3686623 (10Marostegui) [08:56:36] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Decommission db1050 - https://phabricator.wikimedia.org/T178162#3686629 (10Marostegui) a:03Cmjohnson db1050 can now be decommissioned by @Cmjohnson @Cmjohnson remember that one of the disks is failed, so it would be good to identify that one before de... [08:56:54] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Decommission db1050 - https://phabricator.wikimedia.org/T178162#3686644 (10Marostegui) [08:57:26] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Decommission db1050 - https://phabricator.wikimedia.org/T178162#3682923 (10Marostegui) [08:58:54] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3686647 (10Marostegui) [09:01:51] 10DBA, 10Data-Services: labsdb1005's mysql crashed - https://phabricator.wikimedia.org/T178272#3686654 (10jcrespo) [09:09:43] 10DBA, 10Data-Services: labsdb1005's mysql crashed - https://phabricator.wikimedia.org/T178272#3686680 (10jcrespo) There is not much to do, this is memory exhaustion/load coming from templatetiger/external links monitor. [09:11:32] 10DBA, 10Data-Services: labsdb1005's mysql crashed - https://phabricator.wikimedia.org/T178272#3686683 (10jcrespo) The high load, however, continues: https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&panelId=19&fullscreen&orgId=1&var-server=labsdb1005&var-network=bond0&from=now-7d&to=now [09:36:12] 10DBA, 10Cloud-Services, 10Collaboration-Team-Triage, 10Community-Tech-Tool-Labs, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#3686728 (10Dereckson) a:05Dereckson>03None [ Deassigning as not a current... [10:18:09] Hey, Is it okay if I run some cleanups for ores_classification in wikidatawiki [10:18:32] yep [10:28:07] aweomse [10:30:30] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 3 others: Deploy dropping wb_entity_per_page table - https://phabricator.wikimedia.org/T177601#3686942 (10Ladsgroup) [11:22:35] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3687005 (10Marostegui) [11:23:36] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3668277 (10Marostegui) [12:54:43] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3687175 (10Marostegui) [13:01:31] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 12 others: "Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3687197 (10jcrespo) 05Open>03Resolved a:03Bawolff I am... [13:06:29] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3687239 (10jcrespo) [13:06:44] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3687257 (10jcrespo) p:05Triage>03Normal [13:15:35] dbstore1001 seems laterly relatively stable, we will see on wednesday [13:16:09] it tool 3-4 days to perform the backups, it is not scalable [13:17:16] yeah, we need to migrate it to multiinstance [13:37:01] 10DBA, 10Operations, 10Ops-Access-Requests, 10cloud-services-team (Kanban): Access to raw database tables on labsdb* for wmcs-admin users - https://phabricator.wikimedia.org/T178128#3681627 (10jcrespo) This is a problem created by cloud people themselves. I told them to create the view user with localhost... [13:47:32] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3687355 (10jcrespo) ``` shard=s1; mysql -BN -A --socket=/run/mysqld/mysqld.$shard.sock -e "SHOW DATABASES like '%wik%'" | while read db; do echo "$db: "; mysql -BN -A --socket=... [14:26:54] 10DBA, 10Operations, 10ops-codfw: db2081 unreachable - https://phabricator.wikimedia.org/T178140#3687513 (10Papaul) p:05Triage>03Normal [14:45:34] 10DBA, 10InternetArchiveBot, 10Tool-Database-Queries: Not Working - https://phabricator.wikimedia.org/T178294#3687596 (10Cyberpower678) This appears to be a DB problem. This is making the tool and the bot notoriously unreliable. IABot workers are constantly crashing on different wikis due to DB failures. [14:55:03] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3687662 (10jcrespo) ``` shard=s2; mysql -BN -A --socket=/run/mysqld/mysqld.$shard.sock -e "SHOW DATABASES like '%wik%'" | while read db; do echo "$db: "; mysql -BN -A --socket=... [14:58:44] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3687689 (10jcrespo) {P6124, lines=20} [15:04:51] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3687751 (10jcrespo) ``` shard=s4; mysql -BN -A --socket=/run/mysqld/mysqld.$shard.sock -e "SHOW DATABASES like '%wik%'" | while read db; do echo "$db: "; mysql -BN -A --socket=... [15:17:45] 10DBA, 10InternetArchiveBot, 10Tool-Database-Queries: Not Working - https://phabricator.wikimedia.org/T178294#3687804 (10Marostegui) Can you give some more details about this? Where is this happening? Where are you trying to connect to? Which user are you using to connect to the DB? [15:18:07] 10DBA, 10Data-Services, 10Security-Team, 10Patch-For-Review: pagetranslation log_type missing on replicas - https://phabricator.wikimedia.org/T178052#3687806 (10Andrew) 05Open>03Resolved a:03Andrew All set. [15:20:53] 10DBA, 10InternetArchiveBot, 10Tool-Database-Queries: Not Working - https://phabricator.wikimedia.org/T178294#3687826 (10bd808) Likely related to {T178272} [15:21:22] 10DBA, 10Data-Services, 10InternetArchiveBot: Not Working - https://phabricator.wikimedia.org/T178294#3687829 (10bd808) [15:22:34] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687379 (10bd808) [15:22:38] 10DBA, 10Operations, 10ops-codfw: db2081 unreachable - https://phabricator.wikimedia.org/T178140#3687832 (10Papaul) Please see attachment for the error that was on the screen {F10261356} Steps taken: - Removed PSU's for couple of minutes - Update IDRAC firmware from 2.40 to 2.50 - Update BIOS from 2.4.3 t... [15:22:46] 10DBA, 10Data-Services: labsdb1005's mysql crashed - https://phabricator.wikimedia.org/T178272#3686654 (10Marostegui) The high load looks gone (or at least as it used to be before): https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&panelId=19&fullscreen&orgId=1&var-server=labsdb1005&var... [15:24:41] 10DBA, 10Operations, 10ops-codfw: db2081 unreachable - https://phabricator.wikimedia.org/T178140#3687839 (10Marostegui) Thanks @Papaul for troubleshooting this. This was one of the new servers we recently bought. Will you talk to the vendor to get a technician to look at it or advise? This server was not i... [15:25:13] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687843 (10Cyberpower678) >>! In T178294#3687804, @Marostegui wrote: > Can you give some more details about this? > Where is this h... [15:27:11] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687861 (10Marostegui) >>! In T178294#3687843, @Cyberpower678 wrote: >>>! In T178294#3687804, @Marostegui wrote: >> Can you give so... [15:29:42] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687866 (10Cyberpower678) Yes it is still happening. It usually happens when it can't connect to the DB. [15:32:49] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687873 (10Marostegui) >>! In T178294#3687866, @Cyberpower678 wrote: > Yes it is still happening. It usually happens when it can't... [15:33:04] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687379 (10jcrespo) Could it mean someone is abusing the bot usage? Does it depend on recentchanges or user requests? Is there a li... [15:34:54] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687893 (10Cyberpower678) >>! In T178294#3687873, @Marostegui wrote: >>>! In T178294#3687866, @Cyberpower678 wrote: >> Yes it is st... [15:35:13] 10DBA, 10Operations, 10ops-codfw: db2081 unreachable - https://phabricator.wikimedia.org/T178140#3687900 (10Papaul) @Marostegui when you call Dell the first thing they will tell you is to update the firmware that is the reason i didn't call them and i went ahead and update all the firmwares and left the tas... [15:36:08] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687903 (10Marostegui) >>! In T178294#3687893, @Cyberpower678 wrote: > > You'll find those connections aren't constantly sleeping.... [15:37:21] 10DBA, 10Operations, 10ops-codfw: db2081 unreachable - https://phabricator.wikimedia.org/T178140#3682064 (10jcrespo) I think lately we have been doing the following: if it is the first time, upgrade firmware and collect logs/other proof. If it is the second time, collect evidence and ask for replacements. We... [15:37:28] 10DBA, 10Operations, 10ops-codfw: db2081 unreachable - https://phabricator.wikimedia.org/T178140#3687911 (10Marostegui) >>! In T178140#3687900, @Papaul wrote: > @Marostegui when you call Dell the first thing they will tell you is to update the firmware that is the reason i didn't call them and i went ahead a... [15:39:27] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3687922 (10Marostegui) [15:39:32] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687923 (10Cyberpower678) Hmm... IABot doesn't leave connections to idle. Where are they coming from. I do have Workbench open o... [15:40:49] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687924 (10jcrespo) "Usage of connection pools (maintaining open connections without them being in use), persistent connections, or... [15:42:04] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687925 (10Marostegui) It started to work after I removed the limit of 10 connections, so it was clearly trying to create more tha... [15:42:19] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687926 (10Cyberpower678) And I thought it was harder on the server to constantly reconnect when making a query. Where are those t... [15:45:35] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1092 - https://phabricator.wikimedia.org/T177264#3687936 (10Marostegui) @Cmjohnson any update? [15:45:43] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687937 (10jcrespo) > And I thought it was harder on the server to constantly reconnect If this was a dedicated server, pooling wo... [15:46:24] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1092 - https://phabricator.wikimedia.org/T177264#3687939 (10Cmjohnson) @marostegui I received an email over the weekend that the disk has shipped. I expect it today or tomorrow. [15:46:37] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687940 (10Cyberpower678) >>! In T178294#3687925, @Marostegui wrote: > > It started to work after I removed the limit of 10 connec... [15:46:42] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1092 - https://phabricator.wikimedia.org/T177264#3687941 (10Marostegui) Excellent! Thank you! [15:51:37] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687951 (10Cyberpower678) >>! In T178294#3687937, @jcrespo wrote: >> And I thought it was harder on the server to constantly reconn... [15:53:44] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3687953 (10Marostegui) [16:02:02] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687981 (10Cyberpower678) BTW, I'm still getting the errors. [16:03:02] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687983 (10Marostegui) >>! In T178294#3687981, @Cyberpower678 wrote: > BTW, I'm still getting the errors. It works for me everytim... [16:06:02] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3687994 (10Cyberpower678) >>! In T178294#3687983, @Marostegui wrote: >>>! In T178294#3687981, @Cyberpower678 wrote: >> BTW, I'm sti... [16:25:49] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688043 (10jcrespo) We need the mysql error... [16:46:03] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3688087 (10jcrespo) ``` shard=s6; mysql -BN -A --socket=/run/mysqld/mysqld.$shard.sock -e "SHOW DATABASES like '%wik%'" | while read db; do echo "$db: "; mysql -BN -A --socket=... [16:55:24] 10DBA, 10Operations, 10Support-and-Safety, 10Patch-For-Review, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3688097 (10jrbs) >>! In T174370#3685824, @Dereckson wrote: > There is hi.wiktionary to create soon, I'll see next week to plan a window... [17:05:21] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688145 (10Cyberpower678) 2017-10-16 17:01:25: (mod_fastcgi.c.2673) FastCGI-stderr: PHP Warning: mysqli_connect(): (HY000/1226): U... [17:07:05] 10DBA, 10Patch-For-Review, 10codfw-rollout: Replace some masters in eqiad while it is not active - https://phabricator.wikimedia.org/T162133#3688150 (10Cmjohnson) [17:07:07] 10DBA, 10Operations, 10ops-eqiad: Decommission db1022 (Was: db1022 broke while changing topology on s6- evaluate if to fix or directly decommission) - https://phabricator.wikimedia.org/T163778#3688148 (10Cmjohnson) 05Open>03Resolved db1022 is no longer. removed from site.pp, all dns is removed, killed in... [17:07:09] 10DBA, 10Operations, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3688151 (10Cmjohnson) [17:15:47] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3688179 (10jcrespo) ``` shard=s7; mysql -BN -A --socket=/run/mysqld/mysqld.$shard.sock -e "SHOW DATABASES like '%wik%'" | while read db; do echo "$db: "; mysql -BN -A --socket=... [17:16:50] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3688180 (10jcrespo) ``` shard=s5; mysql -BN -A --socket=/run/mysqld/mysqld.$shard.sock -e "SHOW DATABASES like '%wik%'" | while read db; do echo "$db: "; mysql -BN -A --socket=... [17:21:34] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688185 (10Marostegui) >>! In T178294#3688145, @Cyberpower678 wrote: > 2017-10-16 17:01:25: (mod_fastcgi.c.2673) FastCGI-stderr: PH... [17:24:48] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688190 (10Cyberpower678) >>! In T178294#3688185, @Marostegui wrote: >>>! In T178294#3688145, @Cyberpower678 wrote: >> 2017-10-16 1... [17:27:42] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3688203 (10jcrespo) Given the above data, I would like to disable recentchanges from wikidata (temporarily), purge and defragment on the following wikis: ``` bgwiki itwiki svw... [17:31:26] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3688212 (10Ladsgroup) I'm backporting a fix that would help this matter greatly, I don't know how to clean old rows but it will stop adding new entries. Would that be enough? [17:31:42] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688213 (10jcrespo) Cyberpower678 file a ticket to wmcs Data services, and CC me. Do not expect it to be quick. For the time being,... [17:33:33] 10DBA, 10Operations, 10ops-eqiad: decommission db1018 - https://phabricator.wikimedia.org/T176215#3688214 (10Cmjohnson) [17:33:43] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688215 (10bd808) >>! In T178294#3688190, @Cyberpower678 wrote: > Thank you. What do I do to request a dedicated DB instance for C... [17:36:38] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688218 (10Cyberpower678) >>! In T178294#3688213, @jcrespo wrote: > Cyberpower678 file a ticket to wmcs Data services, and CC me. D... [17:41:16] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3688227 (10daniel) @Ladsgroup which fix? @jcrespo what kind of purge do you mean? which rows are being deleted? I'm fine with temporarily disabling change prop, as long as "te... [17:42:19] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3688232 (10Ladsgroup) @daniel Your patch, as @Lydia_Pintscher said should be backported. [17:43:08] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688247 (10jcrespo) > with that and confirmed himself that the load was negligible https://grafana.wikimedia.org/dashboard/file/se... [17:48:35] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3688266 (10jcrespo) @daniel This is the exact ticket than T177772, except with less urgency; as those wikis have less traffic. That doesn't mean they are not as impacted, becau... [17:52:36] 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290#3688272 (10Reedy) >>! In T178290#3688227, @daniel wrote: > Also, it would be nice to have a plan for getting the data that we missed while this was disabled. Is there really m... [18:07:14] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688319 (10Cyberpower678) I do find that somewhat difficult to believe. When the new DB was tested, that was with 28 concurrent co... [18:24:36] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688366 (10Marostegui) Let's see how the graph goes after disabling the limit as we did an hour ago... [18:31:48] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688386 (10bd808) tools.cyberbot (s51059) is currently the number 5 user of disk space on tools.labsdb ([[https://tools.wmflabs.org... [18:44:50] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688455 (10jcrespo) We were commenting this morning the binary_log usage: ```lines=10 SELECT * FROM user_statistics ORDER BY BINL... [19:06:32] 10DBA, 10Data-Services, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3688553 (10russblau) I have found an apparent replication issue on one of the new servers: > tools.dplbot@tools-bastion-03:~$ mysql -h enwiki.analytics.db.svc.eqiad.wmflabs enwiki_p > Reading table info... [19:07:50] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688556 (10Cyberpower678) >>! In T178294#3688386, @bd808 wrote: > tools.cyberbot (s51059) is currently the number 5 user of disk sp... [19:12:11] 10DBA, 10Data-Services, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3688577 (10jcrespo) ``` root@labsdb1009[enwiki]> select page_title, cl_to from page, categorylinks where page_id = cl_from and cl_from=335652; +------------+--------------------------------+ | page_title |... [19:13:04] 10DBA, 10Data-Services, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3688580 (10Marostegui) On a core/production server it has the same result: ``` root@PRODUCTION s1 slave[enwiki]> select page_title, cl_to from page, categorylinks where page_id = cl_from and cl_from=33565... [19:13:16] haha [19:14:53] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688581 (10bd808) >>! In T178294#3688556, @Cyberpower678 wrote: > Hmm...That was the major table I spent the most time optimizing.... [19:16:44] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review: Find test wiki(s) for new description usage and enable there - https://phabricator.wikimedia.org/T177155#3688589 (10hoo) Deployment scheduled for 13:00–14:00 UTC tomorrow (Oct 17). [19:17:14] 10DBA, 10Data-Services, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3688601 (10Cyberpower678) >>! In T178294#3688581, @bd808 wrote: >>>! In T178294#3688556, @Cyberpower678 wrote: >> Hmm...That was th... [19:33:47] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review: Find test wiki(s) for new description usage and enable there - https://phabricator.wikimedia.org/T177155#3688648 (10hoo) [19:41:39] marostegui: jynus if you are around, I'm done cleaning up ores_classification in wikidatawiki, if you shrink it'll probably open up lots of space [19:43:15] thanks Amir1 [19:43:19] can you comment on ticket [19:43:20] ? [19:43:26] sure [19:43:26] so that manuel can see it too tomorow [19:43:33] CC me an manuel [19:43:50] sure [19:45:43] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review: Find test wiki(s) for new description usage and enable there - https://phabricator.wikimedia.org/T177155#3688682 (10hoo) I didn't realize earlier that I can't be around at that time, thus I moved the deploy to 08:00–09:00 UTC Wed... [19:47:52] 10DBA, 10Data-Services, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3688688 (10russblau) User error :-) please disregard. [19:50:19] 10DBA, 10Data-Services, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3688692 (10jcrespo) We should update https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database/Replica_drift and close this ticket, as it is unlikely that there will be drift again (due to ROW + GTID), u... [19:54:14] marostegui: jynus: Can I get a +1 on https://gerrit.wikimedia.org/r/384003 if one of you is around? [19:57:32] it was ok [19:57:45] but I cannot review it right now, I have been working for 12 hours already [20:00:16] oh :/ I'll poke you or Manuel tomorrow [20:02:48] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3688756 (10hoo) I've scheduled the deployment for Wednesday (October 18) 18:00–19:00 UTC. I'll closely monitor the sit... [20:03:46] 10DBA, 10Data-Services, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3688758 (10MusikAnimal) No errors since October 12. I think we can close this. Thank you again for your assistance! :) [20:04:32] 10DBA, 10Data-Services, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3688761 (10jcrespo) 05Open>03Resolved a:03Marostegui [20:06:31] 10DBA, 10Operations, 10Puppet: Switch databases to the future parser - https://phabricator.wikimedia.org/T172498#3688779 (10jcrespo) 05Resolved>03Open Open because I didn't get an answer. [20:22:24] 10DBA, 10Data-Services, 10User-bd808, 10cloud-services-team (Kanban): Create and announce timeline for shutting down labsdb100[13] - https://phabricator.wikimedia.org/T175086#3688820 (10bd808) Drafted on wikitech -- https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown We need to pick dates/... [21:31:28] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 10MW-1.29-release (WMF-deploy-2017-04-25_(1.29.0-wmf.21)), and 5 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3689018 (10Ladsgroup) I've done cleaning up the table in wikidatawiki,... [23:28:11] 10DBA, 10Data-Services, 10Tracking: Wikireplica service for tools and labs - issues and missing available views (tracking) - https://phabricator.wikimedia.org/T150767#2795855 (10bd808) @jcrespo Would you feel ok about closing this tracking bug and instead pointing people to the #data-services tag? If we need... [23:32:15] 10DBA, 10Cloud-VPS, 10Data-Services: Add pp_propname/pp_value index to Labs replica - https://phabricator.wikimedia.org/T140609#3689293 (10bd808) [23:32:21] 10DBA, 10Cloud-VPS, 10Data-Services: rev_len should be available also for deleted revisions in database replicas - https://phabricator.wikimedia.org/T101631#3689296 (10bd808) [23:32:26] 10DBA, 10Cloud-VPS, 10Data-Services: Queries of commonswiki_p.filearchive for fa_sha1 are slow - https://phabricator.wikimedia.org/T71088#3689297 (10bd808)