[00:21:18] 10DBA, 10Cloud-Services, 10Cloud-VPS: Database upgrade MariaDB 10: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127#731912 (10kaldari) FYI, the lack of log indexes on some wikis is causing issues for XTools. See, for example, T171264#3468493. [00:22:14] 10DBA, 10Cloud-Services, 10Cloud-VPS: Database upgrade MariaDB 10: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127#3468673 (10kaldari) p:05Low>03Normal [01:35:40] 10DBA, 10Cloud-Services, 10Cloud-VPS: Database upgrade MariaDB 10: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127#3468760 (10MusikAnimal) Yeah, the `log_title_time` index seems especially helpful when doing things like looking up a block log. You could use `l... [05:13:30] 10DBA, 10Data-Services: Database upgrade MariaDB 10: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127#3468943 (10bd808) [07:08:49] 10DBA, 10Wikidata: Populate term_full_entity_id on test.wikidata.org - https://phabricator.wikimedia.org/T171461#3469011 (10Marostegui) >>! In T171461#3465817, @aude wrote: > i can maybe do this later today / this evening (US time) This wasn't run in the end, no? Just to confirm :) [07:50:18] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#3469042 (10Marostegui) click_tracking has been backuped: ``` root@dbstore1001:/srv/tmp... [08:13:34] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#3469071 (10Marostegui) click_tracking_user_properties was empty in lots of places, but... [09:39:55] 10DBA, 10wikitech.wikimedia.org: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3469274 (10MarcoAurelio) p:05Triage>03Lowest [09:43:16] 10DBA, 10wikitech.wikimedia.org: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3469261 (10Marostegui) I guess you are aware that rename a database isn't something supported on MySQL/MariaDB at the moment. While you can rename a table, a database rename isn't supported yet. [09:56:06] 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3469341 (10Marostegui) Hi, There has been an spike on the pc disk usage again - starting yesterday at around 22:30 UTC - the disk space consumption h... [09:57:54] 10DBA, 10wikitech.wikimedia.org: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3469348 (10MarcoAurelio) 05Open>03stalled Forgot to set it to stalled. Yes, I was not given the exact reason, but I was told that it was not currently possible. Still this is a valid task that... [09:59:02] 10DBA, 10wikitech.wikimedia.org: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3469353 (10Marostegui) There are ways to rename a database, but not using a mysql native command, but exporting and importing into a new database with the correct name etc. It requires downtime, te... [10:08:52] 10DBA, 10wikitech.wikimedia.org: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3469378 (10MarcoAurelio) Doesn't look urgent enough to me to duplicate labswiki and paste it to a new wikitech DB, etc. However I leave that decision to the #cloud-services-team. It was also sugges... [10:10:26] 10DBA, 10cloud-services-team, 10wikitech.wikimedia.org: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3469387 (10Marostegui) [10:12:47] 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3469394 (10Marostegui) p:05High>03Unbreak! [10:13:11] 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3469396 (10Marostegui) >>! In T167784#3447220, @gerritbot wrote: > Change 361659 restored by Jcrespo: > Parsercache: Reduce expiration time to 22 days... [10:25:27] 10DBA, 10Patch-For-Review: Refactor prometheus-mysqld-exporter to support multi-instance hosts - https://phabricator.wikimedia.org/T170666#3469469 (10jcrespo) [10:25:45] 10DBA, 10Patch-For-Review: Refactor prometheus-mysqld-exporter to support multi-instance hosts - https://phabricator.wikimedia.org/T170666#3438730 (10jcrespo) [10:25:47] 10DBA, 10Operations, 10monitoring, 10Patch-For-Review, 10Prometheus-metrics-monitoring: MySQL monitoring with prometheus - https://phabricator.wikimedia.org/T143896#3469475 (10jcrespo) [10:26:06] 10DBA, 10Operations, 10monitoring, 10Patch-For-Review, 10Prometheus-metrics-monitoring: MySQL monitoring with prometheus - https://phabricator.wikimedia.org/T143896#2582458 (10jcrespo) [10:27:46] 10DBA, 10Patch-For-Review: Refactor prometheus-mysqld-exporter to support multi-instance hosts - https://phabricator.wikimedia.org/T170666#3469491 (10jcrespo) This is now technically done, missing "only" the refactor of all previous instances into the new model, and the automatic label creation. [10:28:33] 10DBA, 10Patch-For-Review: Refactor prometheus-mysqld-exporter to support multi-instance hosts - https://phabricator.wikimedia.org/T170666#3469494 (10jcrespo) https://grafana-admin.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=codfw%20prometheus%2Fops&var-server=dbstore2002&var-port=13311 [10:47:56] 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3469555 (10Marostegui) So, we have merged both patches and we are running: ``` /usr/local/bin/mwscript purgeParserCache.php --wiki=aawiki --age=1900800... [11:41:24] 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3469672 (10jcrespo) ```name=rows,lines=20 root@pc1005[parsercache]> pager grep Rows PAGER set to 'grep Rows' root@pc1005[parsercache]> SHOW TABLE STATU... [12:05:46] 10DBA, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Patch-For-Review: WMF ParserCache disk space exhaustion - https://phabricator.wikimedia.org/T167784#3469697 (10jcrespo) p:05Unbreak!>03High Running `ALTER TABLE pc000 ENGINE=InnoDB, LOCK=NONE;`allowed us to defragment on the fly, and reduced the f... [12:10:10] screen -r defragment_pcs on neodymium for the defragmenting [12:11:48] cool [12:11:53] how long you think it will take? [12:12:08] a bit [12:12:12] maybe a day or so [12:12:28] a table only takes 3 minutes [12:12:37] but with the deltes and regular traffic... [12:12:57] also we should probably wait [12:13:08] but I want to test it on codfw first [12:14:00] yeah, codfw is a good testing field [12:14:29] yeah, sql_log_bin_0 [12:14:51] s4 codfw replicas may complain [12:15:13] maybe all of this is some large s4 load [12:23:23] 10DBA, 10Wikidata: Populate term_full_entity_id on test.wikidata.org - https://phabricator.wikimedia.org/T171461#3469736 (10aude) done. (took 35 min... test.wikidata has 74000 items and 37000 properties) [13:01:40] https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&panelId=17&fullscreen&orgId=1&var-server=pc2004&var-network=bond0&from=now-3h&to=now [13:01:50] that huge down i guess were binlogs being deleted? [13:01:56] yes [13:02:03] you cannot count on that [13:02:14] I only did it because the deletes will generate more binlogs [13:02:18] so to mitigate that [13:02:24] yeah yeah [13:02:28] just to be sure it was that :) [13:02:45] do you know what I am going to create [13:02:54] a dashboard for disk space [13:03:30] oh nice [13:04:03] or add one to the prometheus breakdowns [13:29:28] 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, 10TCB-Team, and 3 others: Allow setting the watchlist table to read-only on a per-wiki basis - https://phabricator.wikimedia.org/T160062#3469898 (10Tobi_WMDE_SW) [13:41:13] 10DBA, 10Wikidata: Populate term_full_entity_id on test.wikidata.org - https://phabricator.wikimedia.org/T171461#3469950 (10daniel) @aude As far as I can see, Items on the test site have only one label and description each, right? The live site has 500 times as many items, and about two labels and descriptions... [13:50:44] 10DBA, 103d, 10Multimedia, 10MW-1.30-release, and 3 others: Have search recognise STL files as a new kind of media file ('type:3d' or whatever) - https://phabricator.wikimedia.org/T157348#3469993 (10dr0ptp4kt) Thanks, @Krinkle. ^ @MarkTraceur, for you. [14:14:41] 10DBA, 10Wikidata: Populate term_full_entity_id on test.wikidata.org - https://phabricator.wikimedia.org/T171461#3470167 (10daniel) A totally unrepresentative benchmark on my laptop indicates that populating term_full_entity_id without the full rebuild is about 20x faster. That would bring this down to less th... [14:44:23] This upwards trending should be the one we get from the purging/defragmenting https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&panelId=17&fullscreen&orgId=1&var-server=pc2004&var-network=bond0&from=1500982681239&to=1500993802171 [14:47:41] not bad, around 30G so far [15:10:54] 10DBA, 10cloud-services-team, 10wikitech.wikimedia.org: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3470481 (10bd808) I suppose it could happen as part of moving the database tables from hosting on silver.wikimedia.org to hosting in the s3 cluster ({T167973}), but honestl... [15:16:28] 10DBA, 10cloud-services-team, 10wikitech.wikimedia.org: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3470509 (10Marostegui) neither on DBAs :-) [16:42:30] 10DBA, 10Operations, 10Traffic: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#3187493 (10jcrespo) [17:34:32] 10DBA, 10Mail, 10Operations, 10Patch-For-Review: Setup database for dmarc service - https://phabricator.wikimedia.org/T170158#3471138 (10jcrespo) [17:35:05] 10DBA, 10Mail, 10Operations, 10Patch-For-Review: Setup database for dmarc service - https://phabricator.wikimedia.org/T170158#3420991 (10jcrespo) You showed me a link for the database charset recommend, which one was it @herron ? [17:38:35] 10DBA, 10Mail, 10Operations, 10Patch-For-Review: Setup database for dmarc service - https://phabricator.wikimedia.org/T170158#3471178 (10jcrespo) ``` $ mysql -h m1-master.eqiad.wmnet --skip-ssl -e "SHOW CREATE DATABASE rddmarc" +----------+-------------------------------------------------------------------... [17:38:42] 10DBA, 10Mail, 10Operations, 10Patch-For-Review: Setup database for dmarc service - https://phabricator.wikimedia.org/T170158#3471181 (10jcrespo) [17:56:43] 10DBA, 10Mail, 10Operations, 10Patch-For-Review: Setup database for dmarc service - https://phabricator.wikimedia.org/T170158#3471400 (10jcrespo) [17:58:35] 10DBA, 10Mail, 10Operations, 10Patch-For-Review: Setup database for dmarc service - https://phabricator.wikimedia.org/T170158#3420991 (10jcrespo) 05Open>03Resolved ``` root@diadem:~$ mysql -h m1-master.eqiad.wmnet rddmarc -u rddmarc -p Enter password: Welcome to the MySQL monitor. Commands end with ;... [19:06:59] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1066 - https://phabricator.wikimedia.org/T169448#3398609 (10Cmjohnson) Disk replaced and rebuilding Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Rebuild Firmware state: Online, Spun Up Firmware state: Online, Spun Up Fi... [19:27:37] 10DBA, 10Wikimedia-General-or-Unknown, 10Performance: Clean up skin properties - https://phabricator.wikimedia.org/T171643#3471975 (10demon) [19:35:52] 10DBA, 10Wikimedia-General-or-Unknown, 10Performance: Clean up skin properties - https://phabricator.wikimedia.org/T171643#3472015 (10Jdforrester-WMF) [20:37:42] 10DBA, 10Wikimedia-General-or-Unknown, 10Performance: Clean up skin properties - https://phabricator.wikimedia.org/T171643#3471975 (10matmarex) > The **only** valid values here are monobook, vector, cologneblue, modern and the various minervas (which should be merged, see {T171644}). That's not true. '0' an...