[04:49:03] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) p:05Triage→03Normal a:03Marostegui Yay!! [05:02:37] 10DBA, 10Operations, 10ops-codfw: db2127 memory issues - https://phabricator.wikimedia.org/T233184 (10Marostegui) [05:03:07] 10DBA, 10Operations, 10ops-codfw: db2127 memory issues - https://phabricator.wikimedia.org/T233184 (10Marostegui) p:05Triage→03Normal We are leaving this task opened for a few days to see if the errors get back. [05:06:52] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [05:08:29] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [05:19:17] 10DBA: Decommission db2067.codfw.wmnet - https://phabricator.wikimedia.org/T233185 (10Marostegui) [05:20:17] 10DBA: Decommission db2067.codfw.wmnet - https://phabricator.wikimedia.org/T233185 (10Marostegui) 05Open→03Stalled p:05Triage→03Normal This host is now working as m2 master in codfw, and will be replaced once the new misc hosts are bought. [05:20:19] 10DBA, 10Operations: Decommission db2043-db2069 - https://phabricator.wikimedia.org/T228258 (10Marostegui) [05:21:14] 10DBA, 10Operations: Decommission db2043-db2069 - https://phabricator.wikimedia.org/T228258 (10Marostegui) [05:22:29] 10DBA, 10Operations: Decommission db2055.codfw.wmnet - https://phabricator.wikimedia.org/T233186 (10Marostegui) [05:23:34] 10DBA, 10Operations: Decommission db2055.codfw.wmnet - https://phabricator.wikimedia.org/T233186 (10Marostegui) p:05Triage→03Normal [05:28:35] 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui) [05:29:53] 10DBA, 10Operations, 10Patch-For-Review: Decommission db2055.codfw.wmnet - https://phabricator.wikimedia.org/T233186 (10Marostegui) [05:46:02] 10DBA, 10Operations, 10Patch-For-Review: Decommission db2055.codfw.wmnet - https://phabricator.wikimedia.org/T233186 (10Marostegui) [05:49:48] 10DBA, 10Operations: Decommission db2043-db2069 - https://phabricator.wikimedia.org/T228258 (10Marostegui) [05:53:55] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) I want to alter s6 first host by host, if everything goes fine, I will later do codfw all at once with replication. I am also interested i... [06:01:38] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) >>! In T233135#5502106, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=https://tools.wmflabs.org/s... [06:30:10] 10DBA: Drop frwiki.archive_save table - https://phabricator.wikimedia.org/T233187 (10Marostegui) [06:30:29] 10DBA: Drop frwiki.archive_save table - https://phabricator.wikimedia.org/T233187 (10Marostegui) p:05Triage→03Normal [08:21:57] 10DBA, 10Operations, 10serviceops, 10Goal: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo) [08:42:48] 10DBA, 10Operations, 10serviceops, 10Goal: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts: ` ['backup1001.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-a... [08:52:31] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) s6 db2089:3316 I found this error on the three wikis that live there: `frwiki jawiki ruwiki` ` ERROR 1091 (42000) at line 37: Can't DROP '... [09:01:52] 10Blocked-on-schema-change, 10GlobalBlocking: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 (10Marostegui) [09:04:26] 10DBA, 10Operations, 10serviceops, 10Goal: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['backup1001.eqiad.wmnet'] ` and were **ALL** successful. [09:20:10] 10DBA, 10Operations, 10serviceops, 10Goal: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo) backup1001 was also setup, however there is still a missing disk: T232882#5502241. Separating enclosures into different logical drives is going to pay off earlier... [10:29:42] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) dbproxy1014 has been tested and it is now m1-master. In a couple of days I will revert this change as dbproxy1014 is in a rack that requires maintenance to the PDU... [11:15:21] 10DBA, 10Operations, 10serviceops, 10Goal: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10akosiaris) >>! In T229209#5502350, @jcrespo wrote: > > @akosiaris I would like to disable the accidental reimage of these servers (we suffered from these on a board change... [11:42:04] 10DBA: Decommission dbproxy1006.eqiad.wmnet - https://phabricator.wikimedia.org/T233207 (10Marostegui) [11:42:21] 10DBA: Decommission dbproxy1006.eqiad.wmnet - https://phabricator.wikimedia.org/T233207 (10Marostegui) p:05Triage→03Normal [11:43:07] 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui) [12:02:55] 10DBA: Decommission dbproxy1006.eqiad.wmnet - https://phabricator.wikimedia.org/T233207 (10Marostegui) [12:04:33] 10DBA: Decommission dbproxy1006.eqiad.wmnet - https://phabricator.wikimedia.org/T233207 (10Marostegui) Stopped haproxy to make sure nothing really uses it before decommissioning ` root@dbproxy1006:~# systemctl stop haproxy root@dbproxy1006:~# echo "show stat" | socat /run/haproxy/haproxy.sock stdio 2019/09/18 12... [12:23:59] 10Blocked-on-schema-change, 10GlobalBlocking: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 (10Marostegui) [12:26:34] 10Blocked-on-schema-change, 10GlobalBlocking: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 (10Marostegui) I will start with s6 which I will do on codfw master first, and then on each slave on eqiad. The tables have only 4 rows. If everything goes fine, I... [12:29:26] can I use this in our presentation? https://i.imgflip.com/3avgu4.jpg [12:31:25] hahaha [12:31:27] +1 [12:34:29] hi everybody [12:35:04] I am reading the profile::mariadb::backup::mydumper profile to understand how to configure a regular backup for matomo (matomo1001) and analytics-meta (an-coord1001) [12:35:33] one doubt that I have is about the backups.cnf, since it mentions sections (s1, m4, etc..) [12:35:41] but the analytics databases are not in any [12:35:53] sections are really just replica groups [12:36:04] if you just have one server to backup, just give it a name [12:36:07] is there any setting that you'd prefer me to use? [12:36:09] it will be the name of the backup [12:36:26] "what this backups is" [12:36:42] if it is a backup of matomo, just use matomo [12:36:54] or whatever the database is named [12:38:41] super [12:38:51] it is an identifier you chose [12:39:07] second question is about the backup user - should I set up the 'dump' user in both dbs? [12:39:10] of course, just don't name it enwiki, or you wil lenter in your wanted list :-D [12:39:49] what are the 2 dbs? a master and a replica, or are they independent? [12:40:10] independent, one is on matomo1001, the other one on an-coord1001 [12:40:22] maybe you want to upload a WIP patch and we anc talk about it? [12:40:31] ack, will do :) [12:40:33] that may be easier, just a suggetsion [12:40:47] yep now I know more or less how to proceed [12:40:50] thanks :) [12:40:57] if it is independet [12:41:02] it should be 2 "sections" [12:41:19] e.g. matomo and an-coordinator or whatever name you prefer [12:42:03] note we organize around instances- the tool will generate independent files per database [12:42:29] but I don't know much about your organization, so please send a patch even as a draft [12:42:37] and we can comment there [12:42:51] 10Blocked-on-schema-change, 10GlobalBlocking: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 (10Marostegui) [12:47:03] 10Blocked-on-schema-change, 10GlobalBlocking: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 (10Marostegui) s6 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1005 [x] db1139 [] db1131 [] db1125 [] db1113 [] db1098 [] db109... [13:30:52] 10Blocked-on-schema-change, 10GlobalBlocking: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 (10Marostegui) [13:58:13] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Anomie) >>! In T233135#5502290, @Marostegui wrote: > s6 db2089:3316 > I found this error on the three wikis that live there: `frwiki jawiki ruwiki` >... [14:09:18] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) Thanks - so I will include an `DROP INDEX IF EXISTS usertext_timestamp ` and will also leave the `DROP INDEX IF EXISTS ar_usertext_timesta... [14:53:12] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) There is another issue: for the `logging` table, we are dropping the `log_user` column, which is the one we have the partitions based on:... [15:00:55] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Anomie) No, MediaWiki knows nothing about the paritioning. That's purely a Wikimedia thing. [15:01:50] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) >>! In T233135#5503197, @Anomie wrote: > No, MediaWiki knows nothing about the paritioning. That's purely a Wikimedia thing. Gotcha - so... [15:01:54] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Anomie) Also, see {T223151} where we discussed this before since we knew this was coming. [15:03:27] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) >>! In T233135#5503200, @Anomie wrote: > Also, see {T223151} where we discussed this before since we knew this was coming. Which means we... [15:09:26] 10DBA, 10Core Platform Team Legacy (Watching / External), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Marostegui) So, I guess we'd need to decide what to do with the `logging` table on the special replicas as we are already... [15:19:02] 10DBA, 10Core Platform Team Legacy (Watching / External), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Marostegui) I will try to place a non-partitioned host tomorrow on the logging enwiki section for some time and enable the... [15:19:11] 10DBA, 10Core Platform Team Legacy (Watching / External), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Marostegui) p:05Triage→03Normal [15:20:07] 10DBA, 10Core Platform Team Legacy (Watching / External), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10jcrespo) Two options- the partitioning is no longer needed due to the new schema (preferred) or b) the partitioning is sti... [15:55:14] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Anomie) >>! In T233135#5503075, @Anomie wrote: > WTF is going on there? In case you're curious, T233221 has all the details of WTF is going on there. [17:43:34] 10DBA, 10Analytics, 10Data-Services: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Urbanecm) a:03Marostegui Database was created.