[05:25:10] 10DBA, 10wikitech.wikimedia.org, 10User-notice, 10cloud-services-team (Kanban): Restart m5 master (db1128) - https://phabricator.wikimedia.org/T272388 (10Johan) Tech News might normally be a bit overkill for Wikitech being down for a couple of minutes (I'd recommend wikitech-l and a few short posts to a co... [05:33:39] 10DBA, 10wikitech.wikimedia.org, 10User-notice, 10cloud-services-team (Kanban): Restart m5 master (db1128) - https://phabricator.wikimedia.org/T272388 (10Andrew) @Marostegui that timing sounds fine to me, especially if someone other than me (@Johan?) announces the downtime in advance. [05:42:41] 10DBA, 10wikitech.wikimedia.org, 10User-notice, 10cloud-services-team (Kanban): Restart m5 master (db1128) - https://phabricator.wikimedia.org/T272388 (10Marostegui) >>! In T272388#6760598, @Andrew wrote: > @Marostegui that timing sounds fine to me, especially if someone other than me (@Johan?) announces t... [05:51:33] 10DBA, 10wikitech.wikimedia.org, 10User-notice, 10cloud-services-team (Kanban): Restart m5 master (db1128) - https://phabricator.wikimedia.org/T272388 (10Johan) To be very clear we're talking about the same thing, by "Wikitech" we're just referring to wikitech.wikimedia.org/wiki/ here and nothing else right? [05:52:06] 10DBA, 10wikitech.wikimedia.org, 10User-notice, 10cloud-services-team (Kanban): Restart m5 master (db1128) - https://phabricator.wikimedia.org/T272388 (10Marostegui) Yes, that's it [05:54:10] 10DBA, 10Language-Team, 10MediaWiki-extensions-Translate, 10User-brennen, 10Wikimedia-production-error: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Marostegui) Please, before creating the table, let us know if the table can be replicated... [05:59:02] 10DBA, 10wikitech.wikimedia.org, 10User-notice, 10cloud-services-team (Kanban): Restart m5 master (db1128) - https://phabricator.wikimedia.org/T272388 (10Johan) I think the mailing lists and Tech News can be considered having announced it in advance. Were it a wiki seeing a lot of traffic, or we were expec... [06:02:30] 10DBA, 10MediaWiki-extensions-Translate, 10Language-Team (Language-2021-January-March), 10User-brennen, 10Wikimedia-production-error: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10abi_) a:03abi_ [06:07:47] 10Blocked-on-schema-change, 10DBA: Increase size of slot_roles.role_id - https://phabricator.wikimedia.org/T270054 (10Marostegui) Schema change started on s3 - it will take around 15h to finish. [06:13:46] 10DBA, 10MediaWiki-extensions-Translate, 10Language-Team (Language-2021-January-March), 10User-brennen, 10Wikimedia-production-error: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10abi_) Let me summarise: 1. We added a new table but did not... [06:24:04] 10DBA, 10MediaWiki-extensions-Translate, 10Language-Team (Language-2021-January-March), 10Patch-For-Review, and 2 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Marostegui) @abi_ is the data on that table public? [06:29:35] I am starting percona server on db2102 for testing [07:04:57] host back as mariadb [07:21:26] 10DBA, 10MediaWiki-extensions-Translate, 10Language-Team (Language-2021-January-March), 10Patch-For-Review, and 2 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10abi_) >>! In T272428#6760649, @Marostegui wrote: > @abi_ is the data on th... [07:31:48] 10DBA, 10MediaWiki-extensions-Translate, 10Security-Team, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Marostegui) >>! In T272428#6760710, @abi_ wrote: >>>! In T272428#6760649, @Ma... [07:33:15] 10DBA, 10MediaWiki-extensions-Translate, 10Security-Team, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Nikerabbit) [07:36:23] 10DBA, 10MediaWiki-extensions-Translate, 10Security-Team, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Nikerabbit) We can unblock the train without the creation of this table, so I... [07:36:51] 10DBA, 10MediaWiki-extensions-Translate, 10Security-Team, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10abi_) @Marostegui The patch that fixes this particular issue: [[ https://gerr... [07:44:26] 10DBA, 10MediaWiki-extensions-Translate, 10Security-Team, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Marostegui) >>! In T272428#6760736, @abi_ wrote: > @Marostegui The patch that... [08:11:09] 10DBA, 10MediaWiki-extensions-Translate, 10Security-Team, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10abi_) Related patch is merged. [09:30:28] jynus: Are you using db2102 for anything or can I rebuild enwiki there? [09:30:50] db2102, let me see [09:32:11] not at the moment, I was using db1133 for media backup metadata tests [09:32:28] jynus: so can I delete sqldata/ and rebuild it? [09:32:47] let me double check there is nothing I want to keep quickly [09:32:58] sure, no rush [09:34:28] there is a testing table, but I don't remember what for, so go ahead [09:35:02] thanks! [09:38:02] <_joe_> hi, can you confirm this https://grafana-rw.wikimedia.org/d/000000106/parser-cache?viewPanel=24&orgId=1 is the total SELECT queries per minute to our parsercache cluster? [09:39:28] _joe_: that would be https://grafana.wikimedia.org/d/000000278/mysql-aggregated?viewPanel=8&orgId=1&var-site=eqiad&var-group=parsercache&var-shard=All&var-role=All [09:44:22] <_joe_> so 6k per second? uhm [09:44:30] <_joe_> what is the other number then? [09:45:04] _joe_: keep in mind that that graph are rows per second [09:45:17] queries would be https://grafana.wikimedia.org/d/000000278/mysql-aggregated?viewPanel=1&orgId=1&var-site=eqiad&var-group=parsercache&var-shard=All&var-role=All [09:45:23] let me check the reads [09:45:24] <_joe_> marostegui: actually, what you're showing me is rows per sec, and given we need 2 hits per parsercache object, it fits [09:46:07] _joe_: yes, that is rows per second [09:46:31] <_joe_> I mean it fits with the numbers from the other graph, where it shows 3k req/s [09:46:33] _joe_: https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=16&orgId=1&from=now-24h&to=now&var-server=pc1007&var-port=9104 selects on a host (we have 3, pc1007, pc1008 and pc1009) [09:47:27] <_joe_> marostegui: that's around 3.3k req/s,. summing all those sELECTs [09:47:30] <_joe_> ok thanks :) [09:47:37] _joe_: yep [09:49:10] <_joe_> marostegui: do we store revision id or article id in the parsercache key? [09:49:16] <_joe_> I hope the revid [09:49:19] _joe_: let me check [09:49:31] (I would assume revid) [09:49:35] but let me double check [09:49:40] <_joe_> sorry for all the questions, but I don't want to make stupid assumptions [09:51:18] revid [09:51:36] <_joe_> ack, thanks [09:51:39] And from https://www.mediawiki.org/wiki/Manual:Parser_cache#Types it also looks so: "The ParserCache class caches rendered output (HTML plus associated data) for the latest revision of a page. It serves as a semi-permanent store of a wiki's current content as seen by readers. The ParserCache supports varying keys based on options, and uses a two-tiered system to avoid unnecessary cache fragmentation. [09:51:39] " [09:51:41] _joe_: ^ [09:52:03] <_joe_> I'm not sure what the latter part means [09:53:26] 2 tiered maybe as in memcache + mysql? but probably not, as that has nothing to do with fragmentation [09:54:50] ah, no, it is explained here: https://www.mediawiki.org/wiki/Manual:Parser_cache#Cache_Structure_and_Key_Space [09:59:20] <_joe_> oh so just the cache metadata + data [09:59:52] _joe_: yeah, I can share some of the entries on the table, but it is essentially metadata + the render [10:16:06] <_joe_> jynus: now I'm confused, that part of the doc talks about "page ID", not revid [10:16:28] _joe_: but the first part talks about revision [10:16:42] <_joe_> marostegui: exactly, I'm a bit confused [10:16:46] _joe_, I think you may ask mw people, for me internal structure of pc is like a black box [10:16:54] *want to [10:17:05] One example for this is the FlaggedRevs extension which uses a separate ParserCache to store the rendered output of the "stable" revision of each page [10:17:21] <_joe_> ack [10:17:23] <_joe_> also [10:17:23] _joe_: this is a lot more clarifying though [10:17:37] Following the example above where only the dateformat and userlang options affected the output for the page, the key may look something like page_id!dateformat=default:userlang=ru. Thus any cache lookup with dateformat=default and userlang=ru will hit the same cache entry regardless of the values of the rest of the options, since we know from the information in the first cache tier that they did not affect the output. [10:17:41] so pageid then? :-/ [10:17:59] That is for ParserOutput objects. [10:17:59] <_joe_> so we do get UPDATE queries on pc, I guess [10:18:09] _joe_: no, we get REPLACEs [10:18:16] <_joe_> oh ok [10:18:27] <_joe_> yeah then that's all I needed to know [10:23:17] 10DBA, 10MediaWiki-extensions-Translate, 10Security-Team, 10Language-Team (Language-2021-January-March), and 2 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Nikerabbit) p:05Unbreak!→03High To train conductor: A fix has been backpo... [10:41:23] 10DBA, 10cloud-services-team (Kanban): Move wikireplicas under the new sanitarium hosts (db1154, db1155) - https://phabricator.wikimedia.org/T272008 (10Marostegui) [10:41:46] 10DBA, 10cloud-services-team (Kanban): Move wikireplicas under the new sanitarium hosts (db1154, db1155) - https://phabricator.wikimedia.org/T272008 (10Marostegui) clouddb1014:3317 and clouddb1018:3317 moved. [11:07:11] 10DBA, 10MediaWiki-extensions-Translate, 10Security-Team, 10Language-Team (Language-2021-January-March), and 2 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Urbanecm) >>! In T272428#6761132, @Nikerabbit wrote: > To train conductor: A... [11:20:30] 10DBA, 10MediaWiki-extensions-Translate, 10Security-Team, 10Language-Team (Language-2021-January-March), and 2 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Ladsgroup) Yes. To quote https://wikitech.wikimedia.org/wiki/How_to_deploy_co... [11:22:02] 10DBA, 10MediaWiki-extensions-Translate, 10Security-Team, 10Language-Team (Language-2021-January-March), and 2 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10RhinosF1) Is this not the second time in 2 weeks this has happened with WMF b... [13:04:36] 10DBA, 10MediaWiki-extensions-Translate, 10Security-Team, 10Language-Team (Language-2021-January-March), and 2 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Nikerabbit) My apologies. I thought it's not a problem if 1.36.0-wmf.27 is no... [13:44:12] 10DBA, 10MediaWiki-extensions-Translate, 10Security-Team, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Nikerabbit) 05Open→03Resolved Deployed and tested on test.wikipedia.org w... [14:23:39] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host on candidate masters - https://phabricator.wikimedia.org/T271106 (10Kormat) [14:34:33] 10DBA, 10wikitech.wikimedia.org, 10User-notice, 10cloud-services-team (Kanban): Restart m5 master (db1128) - https://phabricator.wikimedia.org/T272388 (10Andrew) >>! In T272388#6760633, @Johan wrote: > I think the mailing lists and Tech News can be considered having announced it in advance. Were it a wiki... [14:38:36] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host on candidate masters - https://phabricator.wikimedia.org/T271106 (10Kormat) [15:15:24] 10DBA, 10MediaWiki-extensions-Translate, 10Security-Team, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10brennen) Thanks all for your assistance. Will roll the train forward to group... [15:26:41] 10DBA, 10MediaWiki-extensions-Translate, 10Security-Team, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10abi_) Thanks >>! In T272428#6761750, @Nikerabbit wrote: > Deployed and teste... [16:03:20] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host on candidate masters - https://phabricator.wikimedia.org/T271106 (10Kormat) [16:41:57] 10Data-Persistence-Backup, 10Analytics: Matomo database backup size doubled, we should check this is normal operation - https://phabricator.wikimedia.org/T272344 (10razzi) @jcrespo It looks like this is normal - traffic to wikimediafoundation.org has spiked since the 20th birthday last week, so the access logs... [16:44:33] 10Data-Persistence-Backup, 10Analytics: Matomo database backup size doubled, we should check this is normal operation - https://phabricator.wikimedia.org/T272344 (10jcrespo) 05Open→03Resolved a:03razzi Cool thanks. I initially filed this because I had missinterpreted the data as the data shrinking (not g... [16:59:03] 10DBA, 10MediaWiki-extensions-Translate, 10Privacy Engineering, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10sbassett) >>! In T272428#6760729, @Marostegui wrote: > @abi_ thanks for... [17:08:39] 10Blocked-on-schema-change: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Ladsgroup) [17:10:31] 10Blocked-on-schema-change: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Ladsgroup) [17:15:21] 10Blocked-on-schema-change: Alter objectcache.exptime - https://phabricator.wikimedia.org/T272512 (10Ladsgroup) [17:18:47] 10DBA, 10MediaWiki-extensions-Translate, 10Privacy Engineering, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Marostegui) Thanks @sbassett - we can definitely filter the table, but... [17:21:23] 10DBA, 10MediaWiki-extensions-Translate, 10Privacy Engineering, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Reedy) FWIW, Translate is only used on circa ~40 wikis. [17:40:13] 10DBA, 10MediaWiki-extensions-Translate, 10Privacy Engineering, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Marostegui) Ah, that's good to know - thanks!. Still, let's try to come... [17:47:26] 10DBA, 10MediaWiki-extensions-Translate, 10Privacy Engineering, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Reedy) And TBH, if it's just a cache, there is probably little to no va... [18:00:11] 10DBA, 10MediaWiki-extensions-Translate, 10Privacy Engineering, 10Language-Team (Language-2021-January-March), and 3 others: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist - https://phabricator.wikimedia.org/T272428 (10Marostegui) Agreed! As I mentioned at T272428#6760729 there's probably... [18:26:15] 10DBA, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10Cmjohnson) the issue I ran into is db1169 was created in netbox w/out a mgmt ip. I didn't see until I went through and assigned mgmt IP's. so now everything is 1 off, I... [18:46:02] 10DBA, 10SRE, 10ops-eqiad: Memory errors on clouddb1019 - https://phabricator.wikimedia.org/T272125 (10Cmjohnson) Record: 1 Date/Time: 08/31/2020 17:37:02 Source: system Severity: Ok Description: Log cleared. ------------------------------------------------------------------------------- Recor... [18:46:41] 10DBA, 10SRE, 10ops-eqiad: Memory errors on clouddb1019 - https://phabricator.wikimedia.org/T272125 (10Cmjohnson) Swapped DIMM A4 with DIMM B4, cleared the system log and powered on. Let's see if the error returns, stays the same or changes. [19:04:48] 10DBA, 10SRE, 10ops-eqiad: Memory errors on clouddb1019 - https://phabricator.wikimedia.org/T272125 (10Cmjohnson) Fast response by the server, after swapping the DIMM, the server was stuck in a continuous reboot. connected the console and see that the server is failing during post at the memory check. Not s... [19:35:09] 10DBA: Grant "sockpuppet_import" user INDEX on "sockpuppet" database - https://phabricator.wikimedia.org/T272533 (10hnowlan) [19:44:06] 10DBA: Grant "sockpuppet_import" user INDEX on "sockpuppet" database - https://phabricator.wikimedia.org/T272533 (10LSobanski) p:05Triage→03Medium [19:46:45] 10DBA: Grant "sockpuppet_import" user INDEX on "sockpuppet" database - https://phabricator.wikimedia.org/T272533 (10WDoranWMF) p:05Medium→03High @LSobanski I'm raising this to high priority as this is a blocker for us to debug a significant issue for us. Let me know if that is not reasonable. [20:36:10] 10DBA, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10Cmjohnson) [20:37:28] 10DBA, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10Cmjohnson) a:05Jclark-ctr→03RobH All of the servers are in the racks, idracs are setup including db1169 and db1175. Outstanding items that @robh will do - raid - p...