[01:12:18] <wikibugs>	 10DBA, 10Reading-Infrastructure-Team-Backlog, 10Release-Engineering-Team-TODO: Drop DB tables for now-deleted zerowiki from production - https://phabricator.wikimedia.org/T227717 (10MaxSem) ==Privacy review== Database tables: ` MariaDB [zerowiki]> show tables; +--------------------------+ | Tables_in_zerowik...
[02:03:32] <wikibugs>	 10DBA, 10Core Platform Team, 10MediaWiki-Database, 10Performance-Team, 10Performance: Obsessive serverIsReadOnly() checking in MySQL - https://phabricator.wikimedia.org/T227838 (10Smalyshev)
[02:03:59] <wikibugs>	 10DBA, 10Core Platform Team, 10MediaWiki-Database, 10Performance-Team, 10Performance: Obsessive serverIsReadOnly() checking in MySQL - https://phabricator.wikimedia.org/T227838 (10Smalyshev)
[06:37:06] <wikibugs>	 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10jcrespo)
[06:46:29] <wikibugs>	 10DBA, 10Core Platform Team, 10MediaWiki-Database, 10Performance-Team, 10Performance: Obsessive serverIsReadOnly() checking in MySQL - https://phabricator.wikimedia.org/T227838 (10jcrespo) May I ask where you tested this, and if it was on your own installation, more data about it (version, topology, conf...
[06:54:31] <wikibugs>	 10DBA, 10MediaWiki-API, 10Performance, 10User-Marostegui: list=logevents slow for users with last log action long time ago - https://phabricator.wikimedia.org/T71222 (10jcrespo) @Anomie it is probably the old decision of Mediawiki vs Wikimedia. I don't have visibility of what is the impact outside of WMF,...
[06:58:29] <wikibugs>	 10DBA, 10Epic: Setup es4 and es5 replica sets for new read-write external store service - https://phabricator.wikimedia.org/T226704 (10jcrespo) > If I recall correctly, the maintenance was along the lines of running MediaWiki's recompression scripts and other similar scripts, which would first require making s...
[07:00:31] <wikibugs>	 10DBA, 10Analytics, 10Analytics-EventLogging, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10jcrespo) Chris, you will need to coordinate with @elukey principally, as he is the person in touch directly with users affected to...
[07:03:14] <wikibugs>	 10DBA, 10Analytics, 10Analytics-EventLogging, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10elukey) We can do it anytime with 10/15 mins of heads up Chris (I need to stop replication and traffic to db1107 before you can op...
[07:05:55] <wikibugs>	 10DBA, 10MediaWiki-API, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Workboards (Clinic Duty Team), 10Wikimedia-production-error: Certain ApiQueryRecentChanges::run api query is too slow, slowing down dewiki - https://phabricator.wikimedia.org/T149077...
[07:21:34] <wikibugs>	 10DBA, 10Reading-Infrastructure-Team-Backlog, 10Release-Engineering-Team-TODO: Drop DB tables for now-deleted zerowiki from production - https://phabricator.wikimedia.org/T227717 (10jcrespo) > I'd say it's safe to store a backup  You probably weren't answering me, but safe on my previous comment was meant no...
[07:53:16] <wikibugs>	 10DBA, 10Core Platform Team, 10MediaWiki-Database, 10Performance-Team, 10Performance: Obsessive serverIsReadOnly() checking in MySQL - https://phabricator.wikimedia.org/T227838 (10Smalyshev) > May I ask where you tested this  On a VM (specifically federated-commons.eqiad.wmflabs)  > and if it was on your...
[09:48:12] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Daimona) 05Stalled→03Open wmf.13, unstalling
[10:06:38] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2045 failed battery - https://phabricator.wikimedia.org/T227862 (10jcrespo)
[10:07:00] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2045 failed battery - https://phabricator.wikimedia.org/T227862 (10jcrespo) I will try to force a relearn process/reboot, in case that works.
[10:14:31] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2045 failed battery - https://phabricator.wikimedia.org/T227862 (10jcrespo) Based on https://noc.wikimedia.org/conf/highlight.php?file=db-codfw.php&1 and T184888 I will switchover codfw master to db2069.
[10:27:33] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2045 failed battery - https://phabricator.wikimedia.org/T227862 (10jcrespo) Everything went well except: ` Updating tendril... [WARNING] Old master not found on tendril server list Updating zarcillo... [WARNING] Old master not found on zarcillo master list `
[10:27:50] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2045 failed battery - https://phabricator.wikimedia.org/T227862 (10jcrespo) p:05Triage→03Normal
[11:02:17] <wikibugs>	 10DBA, 10Core Platform Team, 10MediaWiki-Database, 10Performance-Team, 10Performance: Obsessive serverIsReadOnly() checking in MySQL - https://phabricator.wikimedia.org/T227838 (10jcrespo) Based on your feedback, my guess is that because you are reading from the "master" (there is no other host, really),...
[11:13:26] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2045 failed battery - https://phabricator.wikimedia.org/T227862 (10jcrespo) >>! In T227862#5327825, @jcrespo wrote: > Everything went well except: > ` > Updating tendril... > [WARNING] Old master not found on tendril server list > Updating zarcillo... > [WARNING] Old maste...
[11:16:08] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: db2045 failed battery - https://phabricator.wikimedia.org/T227862 (10jcrespo) ` root@db1115.eqiad.wmnet[zarcillo]> update masters set instance='db2069' where section='x1' and dc='codfw'; Query OK, 1 row affected (0.00 sec) Rows matched: 1  Changed: 1  Warnings: 0  root@db111...
[13:46:02] <wikibugs>	 10DBA, 10Epic, 10Tracking-Neverending: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Physikerwelt)
[13:54:12] <wikibugs>	 10DBA, 10Epic: Setup es4 and es5 replica sets for new read-write external store service - https://phabricator.wikimedia.org/T226704 (10Anomie) No problem, it doesn't hurt to check. At one point we //were// considering the possibility of having the scripts update existing ES entries in place, but in the end we...
[13:54:59] <wikibugs>	 10DBA, 10Epic: Setup es4 and es5 replica sets for new read-write external store service - https://phabricator.wikimedia.org/T226704 (10jcrespo) Thanks, that was probably it. Thanks for clarifying it!
[14:07:20] <wikibugs>	 10DBA, 10Math: Remove table `math` from the database - https://phabricator.wikimedia.org/T196055 (10jcrespo) 05Declined→03Open a:03jcrespo Thanks, if you allow me- I will copy that and reopen so this is still tracked. And maybe I can check or delete on a low traffic db and see what are the results. If yo...
[14:07:29] <wikibugs>	 10DBA, 10Epic, 10Tracking-Neverending: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10jcrespo)
[14:09:23] <wikibugs>	 10DBA, 10Math: Remove table `math` from the database - https://phabricator.wikimedia.org/T196055 (10jcrespo)
[14:11:39] <wikibugs>	 10DBA, 10Math: Remove table `math` from the database - https://phabricator.wikimedia.org/T196055 (10jcrespo) See, we had a bit of a communication loss here, but we may be able to move forward :-D Each person only has a limited amount of information. I will give a look and do some tests and (if you can) I may a...
[14:12:55] <wikibugs>	 10DBA, 10Math: Remove table `math` from the database - https://phabricator.wikimedia.org/T196055 (10Physikerwelt) @jcrespo it would be sufficent if you keep a backup of the table somewhere. The outputhash column can not be reconstructed. Only with that one could find all the old images (if they have not alread...
[14:15:16] <wikibugs>	 10DBA, 10Math: Remove table `math` from the database - https://phabricator.wikimedia.org/T196055 (10jcrespo) Thanks, that is great info, I will come back with a list of images and/or a dump at the beginning of next week.
[14:22:11] <wikibugs>	 10DBA, 10Math: Remove table `math` from the database - https://phabricator.wikimedia.org/T196055 (10Physikerwelt) Thank you.
[22:17:12] <wikibugs>	 10DBA, 10Core Platform Team, 10MediaWiki-Database, 10Performance-Team, 10Performance: Obsessive serverIsReadOnly() checking in MySQL - https://phabricator.wikimedia.org/T227838 (10Smalyshev) > Probably this doesn't happen on WMF production  That's good to know. But I think still looks weird and may be so...
[22:18:23] <wikibugs>	 10DBA, 10Core Platform Team, 10MediaWiki-Database, 10Performance-Team, 10Performance: Obsessive serverIsReadOnly() checking in MySQL - https://phabricator.wikimedia.org/T227838 (10aaron) Is $wgMainCacheType set to CACHE_NONE ?
[22:20:43] <wikibugs>	 10DBA, 10Core Platform Team, 10MediaWiki-Database, 10Performance-Team, 10Performance: Obsessive serverIsReadOnly() checking in MySQL - https://phabricator.wikimedia.org/T227838 (10Smalyshev) > Is $wgMainCacheType set to CACHE_NONE ?  Yes, looks like it. Should it be changed? What would be the proper sett...