[04:37:21] 10DBA, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: Wikibase\Repo\Store\Sql\SqlEntitiesWithoutTermFinder::getEntitiesWithoutTerm can take 19 hours to execute and it is run by the web requests user - https://phabricator.wikimedia.org/T160887#3139193 (10aude) For just items: queries for selecting item... [04:46:15] 10DBA, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: [Task] Remove "all" option for Special:EntitiesWithout*" - https://phabricator.wikimedia.org/T161631#3137680 (10aude) error I get with https://gerrit.wikimedia.org/r/#/c/345186/ when going to https://wikidatawiki/wiki/Special:EntitiesWithoutDescript... [06:13:22] 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3139278 (10Marostegui) So far db1089 and db1092, the hosts that are running the tables with the new PKs in... [07:24:58] 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3139409 (10Marostegui) s6 (frwiki): ``` categorylinks UNIQUE KEY `cl_from` (`cl_from`,`cl_to`), UNIQUE... [08:05:49] 10DBA: Remove partitioning from db2019 (codfw master) commonswiki.templatelinks - https://phabricator.wikimedia.org/T161683#3139513 (10Marostegui) [08:05:56] 10DBA: Remove partitioning from db2019 (codfw master) commonswiki.templatelinks - https://phabricator.wikimedia.org/T161683#3139525 (10Marostegui) p:05Triage>03Normal [08:24:29] 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: *_minor_mime are varbinary(32) on WMF sites, out of sync with varbinary(100) in MW core - https://phabricator.wikimedia.org/T73563#3139561 (10Marostegui) Will alter db2019 tomorrow morning, as yesterday I got stuck in other things and I didn't want to st... [08:34:17] 10DBA, 10MediaWiki-Database: Apply recentchanges special partitioning to a redundant slave at least on s1, s4, and s5 - https://phabricator.wikimedia.org/T120513#3139564 (10jcrespo) 05Open>03Resolved a:03jcrespo This is now done: https://noc.wikimedia.org/conf/highlight.php?file=db-eqiad.php [09:37:01] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: [Task] Remove "all" option for Special:EntitiesWithout*" - https://phabricator.wikimedia.org/T161631#3139617 (10thiemowmde) [09:41:52] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: [Task] Remove "all" option for Special:EntitiesWithout*" - https://phabricator.wikimedia.org/T161631#3137680 (10thiemowmde) a:03daniel [09:56:58] 10DBA, 13Patch-For-Review: run pt-table-checksum on s2 (WAS: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038) - https://phabricator.wikimedia.org/T154485#3139674 (10jcrespo) Checking text now, at least on the master, dbstore1002 and the hosts to be decommissioned: ``` $ while read db;... [09:59:26] ^ \o/ [10:08:15] can I ask to put on the description where the checksums table is, I could not find the one for s6 (it wasn't on ops) [10:08:46] Ah sure [10:08:52] it should be on every schema [10:08:58] as we decided it was easier to scan it later [10:09:02] instead of all the stuff at once [10:09:10] so it should be on frwiki,jawiki and ruwiki [10:09:21] (it is pretty worrying that I know the databases per shard already) [10:10:59] 10DBA, 13Patch-For-Review: run pt-tablechecksum on s6 - https://phabricator.wikimedia.org/T160509#3139732 (10Marostegui) [10:12:15] 10DBA: run pt-tablechecksum on s5 - https://phabricator.wikimedia.org/T161294#3139735 (10Marostegui) [10:13:06] thabks [10:13:30] sorry for the hassle! [10:40:04] jynus, marostegui: if you're not in the middle of something critical, can I "broke" tendril for a minute switching it to codfw view and back to eqiad? (testing for the switchdc) [10:40:15] fine by me :) [10:44:49] jaime? :) [10:46:38] I have saved "select name, master_id from shards;" just in case [10:47:29] that also makes me wondering... should I change also es1 in tendril? not sure if it has any effect [10:47:41] given that they are not replicated or shown in the tree [10:58:23] es1 is now showed [10:58:33] and does not have replication [10:58:44] so not worth doing nothing there [10:59:03] please go ahead [10:59:09] ok thanks [10:59:30] that will serve us to check all codfw->eqiad replication channels work as indeded [10:59:49] * volans running it now [10:59:58] or not :D [11:00:13] because of stupidity [11:00:37] make sure you do not change anything else [11:00:37] 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3139833 (10Marostegui) s6 (frwiki,jawiki,ruwiki) on db1093 has no PK on all the used tables but: ``` chan... [11:01:01] yeah! [11:02:57] fix coming, zuul time and I'll test it for real, sorry for the false test [11:08:17] ready to test again, hopefully for real this time [11:09:41] what the hell, this patch is really unlucky [11:09:49] are you running thing with mysql --skip-ssl ? [11:10:04] yes, mysql --skip-ssl --skip-column-names --batch -e [11:10:13] you shouldn't [11:10:21] because of localhost? [11:10:26] ah [11:10:28] yes [11:10:30] sorry [11:10:53] I am thinking all the time about running from remote [11:11:09] what are the issues? [11:11:15] missing space [11:11:17] :( [11:11:59] the previous one was not actually using the new database parameter [11:12:43] and this just reformatting stupidity when I broke it into 2 lines because of line max lenght limit in tox/flake8 [11:12:46] https://gerrit.wikimedia.org/r/#/c/345308/1/switchdc/stages/t09_tendril.py [11:13:00] now you understand my suffering [11:13:22] yeah, but it's alreay 120 chars... [11:14:17] ok, ready for take 3 [11:15:25] jynus, marostegui done [11:15:55] if you want to take a look [11:16:08] I'm ready to switch it back to eqiad when you want [11:16:42] 10DBA, 13Patch-For-Review: run pt-table-checksum on s2 (WAS: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038) - https://phabricator.wikimedia.org/T154485#3139884 (10jcrespo) I have to do a second pass for : idwiki itwiki nlwiki nowiki plwiki ptwiki svwiki trwiki zhwiki All others had... [11:16:56] checking [11:17:55] marostegui, do you miss any server/repplication channel? All look good to me [11:18:05] I am checking [11:18:20] to me too [11:18:27] this reminds me that we have to double check all codfw masters are on statement [11:18:44] right! [11:18:50] it looks good to me as well [11:18:53] it wouldn't be a huge loss, however [11:19:03] we are taking out that dependency soon [11:19:07] ok, switching back [11:19:08] thanks to the new labs [11:19:17] oh yeah! that's great [11:19:52] done [11:20:34] we should make dbstore2001 depend from db2033 [11:20:37] the master_id matches the one of before the switch [11:20:41] but not a huge priority [11:21:26] we should ask for a spare machine for an x1 slave [11:23:00] volans let me steam some of your time [11:23:03] *steak [11:23:06] *steal [11:23:17] lol [11:23:20] sure [11:23:22] if I want 'R:Class = Role::Mariadb::Groups and R:Class%mysql_group = core and R:Class%mysql_role = master' [11:23:29] but only on one dc? [11:23:58] is there something bundled, do I use the current dc puppet var or a regex on the host? [11:24:08] what is your "expert" opinion [11:24:13] so this is one the limitation that will require us to expand the grammar a bit, I thik we'll cover this on friday, for now in your case you can just use a trick [11:24:20] ok [11:24:21] add: "and db2*" [11:24:24] I just need a hack [11:24:28] thanks :-) [11:24:58] I can explain the detail if you have time/want of course, but first the practical answer ;) [11:25:12] well, I can see why it is difficult [11:25:23] is there something beyond that? [11:25:49] but those things could be workarounded by putting "grains" back on base [11:25:55] ^? [11:26:01] 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3139899 (10Marostegui) I am going to start altering s7 on db1094. I will post the list of tables later, as... [11:26:13] there are 2 limitations, one is that $::site is not actually in puppetdb right now, there is a patch to somehow add it, and the other one is a joint limitation of cumin/puppetdb API that don't allow to mix resource and fact queries right now [11:26:31] oh [11:26:48] well, as I said, we can add variables on base as a quick hack [11:26:52] puppetdb API don't allow it and cumin right now is not able to make multiple queries and merge them, that's the TODO that to make it general will require a bit of change in the grammar [11:26:56] they should be there in the first place [11:27:09] but there seemed to be a salt limitation on install [11:27:09] to encapsulate the current grammar in blocks that can be composed [11:27:17] which probably cumin wouldn't have [11:27:57] also another hack is "and *.codfw.wmnet" [11:28:07] yes [11:28:12] I do not see that as a hack [11:28:13] as long as you don't have hosts .wikimedia.org it works [11:28:22] ah, I see [11:28:28] for those it doesn't :( [11:28:34] maybe even some kind of trick with the ip? [11:28:39] but again, that is a fact [11:28:52] not a variable [11:29:53] "and db2* or es2*" would work? [11:30:02] "and (db2* or es2*)" maybe? [11:31:10] All good, BTW: https://phabricator.wikimedia.org/P5155 [11:31:27] why not just and *.codfw.wmnet? [11:31:29] quicker :D [11:31:46] I had already executed the first one [11:31:54] but would the or work? [11:32:31] 'R:Class = Role::Mariadb::Groups and R:Class%mysql_group = core and R:Class%mysql_role = master and (db2* or es2*)' [11:32:34] matches [11:32:36] db[2016-2019,2023,2028-2029,2033].codfw.wmnet,es[2016,2018].codfw.wmnet [11:32:46] yes, that works [11:53:01] 10DBA, 10Wikidata, 07Performance, 15User-Daniel, and 2 others: Use redis-based lock manager in dispatch changes in production - https://phabricator.wikimedia.org/T159826#3140017 (10Ladsgroup) [11:53:03] 10DBA, 10Wikidata, 07Performance, 15User-Daniel, and 2 others: Use redis-based lock manager in dispatch changes in beta cluster - https://phabricator.wikimedia.org/T159828#3140016 (10Ladsgroup) 05Open>03Invalid [11:53:13] 10DBA, 10Wikidata, 07Performance, 15User-Daniel, and 2 others: Use redis-based lock manager in dispatch changes in beta cluster - https://phabricator.wikimedia.org/T159828#3079937 (10Ladsgroup) Beta cluser doesn't have dispatching. [12:59:07] 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3140135 (10Marostegui) s7 (all wikis but metawiki and centraauth) have been altered and they all have PK b... [13:44:59] 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3140267 (10Marostegui) >>! In T17441#3140135, @Marostegui wrote: > s7 (all wikis but metawiki and centraau... [14:01:58] 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3140319 (10Marostegui) db1091 (s4) is back in the pool with the following tables altered: ``` categorylink... [15:09:44] 10DBA, 10Wikidata, 07Performance, 15User-Daniel, and 2 others: Use redis-based lock manager in dispatch changes in production - https://phabricator.wikimedia.org/T159826#3140531 (10daniel) [15:10:13] 10DBA, 10Wikidata, 15User-Daniel, 15User-Ladsgroup, 03Wikidata-Sprint: Use redis-based lock manager for dispatchChanges on test sites. - https://phabricator.wikimedia.org/T159828#3079937 (10daniel) @Ladsgroup thanks for pointing that out. Let's do this on test.wikidata.org, then. [15:14:14] 10DBA, 06Operations, 10ops-codfw, 10procurement: Adquire temporary box for x1 failover (spare available?) - https://phabricator.wikimedia.org/T161712#3140543 (10jcrespo) [15:14:34] 10DBA, 06Operations, 10ops-codfw, 10procurement: Adquire temporary box for x1 failover on codfw (spare available?) - https://phabricator.wikimedia.org/T161712#3140558 (10jcrespo) [15:19:32] 10DBA, 06Operations, 10ops-codfw, 10procurement: Aquire temporary box for x1 failover on codfw (spare available?) - https://phabricator.wikimedia.org/T161712#3140573 (10Reedy) [15:20:07] 10DBA, 06Operations, 10ops-codfw, 10procurement: Aquire temporary box for x1 failover on codfw (spare available?) - https://phabricator.wikimedia.org/T161712#3140576 (10jcrespo) I think we could use temporarily es2002, but asking first in case there is a more suitable machine available. [15:21:10] 10DBA, 06Operations, 10ops-codfw, 10procurement: Acquire temporary box for x1 failover on codfw (spare available?) - https://phabricator.wikimedia.org/T161712#3140583 (10jcrespo) [15:22:32] marostegui, are you on top of the analytics stuff, or do you need me? [15:23:29] not sure I am following you [15:23:42] the eventlogging maintenance, is that today? [15:23:54] no, that was supposed to happen last week [15:23:57] and it did happen as far as I know [15:24:13] the tables were renamed [15:24:52] you talking about: https://phabricator.wikimedia.org/T160454 ? [15:25:21] yes [15:25:27] sorry, I didn't know that [15:26:04] I saw an email saying "Friendly remainder that this will happen within the next couple hours today." [15:26:09] mmm [15:26:16] That is strange, as I saw the tables renamed last week [15:26:27] maybe it is something else, so discard it if so [15:26:44] I will ping in the ticket [15:28:03] 10DBA, 06Analytics-Kanban, 13Patch-For-Review: Change length of userAgent column on EL tables - https://phabricator.wikimedia.org/T160454#3140604 (10Marostegui) Hi! What's the status of this? [15:31:43] 10DBA, 06Operations, 10ops-codfw, 10procurement: Acquire temporary box for x1 failover on codfw (spare available?) - https://phabricator.wikimedia.org/T161712#3140609 (10jcrespo) I think that is the right term, sorry, I had to look it up in the dictionary. English is not my strong point as a non-native-spe... [15:33:35] 10DBA, 06Analytics-Kanban, 13Patch-For-Review: Change length of userAgent column on EL tables - https://phabricator.wikimedia.org/T160454#3140617 (10Ottomata) Happening today! :) [15:34:36] 10DBA, 10Wikidata, 15User-Daniel, 15User-Ladsgroup, 03Wikidata-Sprint: Use redis-based lock manager for dispatchChanges on test sites. - https://phabricator.wikimedia.org/T159828#3140619 (10jcrespo) Sorry about this- you are not the only "sufferers" of beta not being a reliable place for testing in a tru... [15:39:22] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1067 - https://phabricator.wikimedia.org/T161600#3140623 (10Cmjohnson) @marostegui disk replaced...rebuilding now nclosure Device ID: 32 Slot Number: 11 Drive's position: DiskGroup: 0, Span: 1, Arm: 5 Enclosure position: 1 Device Id: 11 WWN: 5000039788210... [15:39:53] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1067 - https://phabricator.wikimedia.org/T161600#3136638 (10jcrespo) Thank you very much! :-) [15:40:01] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1067 - https://phabricator.wikimedia.org/T161600#3140626 (10Marostegui) Thanks Chris, will monitor it and close the ticket when it's finished [15:40:09] gah! jaime is always faster! :p [15:42:42] 10DBA, 06Operations, 10ops-codfw, 10procurement: codfw: (1) spare pool system for temp allocation as database failover - https://phabricator.wikimedia.org/T161712#3140657 (10RobH) [15:48:36] 10DBA, 06Operations, 10ops-codfw, 10procurement: codfw: (1) spare pool system for temp allocation as database failover - https://phabricator.wikimedia.org/T161712#3140666 (10jcrespo) [15:52:56] 10DBA, 06Operations, 10ops-codfw, 10procurement: codfw: (1) spare pool system for temp allocation as database failover - https://phabricator.wikimedia.org/T161712#3140668 (10RobH) a:03faidon So, for this I have 3 spare machines in codfw. One of them is being used to restore graphite data (so short term... [15:53:26] 10DBA, 06Operations, 10hardware-requests, 10ops-codfw: codfw: (1) spare pool system for temp allocation as database failover - https://phabricator.wikimedia.org/T161712#3140685 (10RobH) [15:55:21] 10DBA, 06Operations, 10hardware-requests, 10ops-codfw: codfw: (1) spare pool system for temp allocation as database failover - https://phabricator.wikimedia.org/T161712#3140695 (10faidon) That's totally fine, approved. [16:00:31] 10DBA, 10Wikidata, 13Patch-For-Review, 15User-Daniel, and 2 others: Use redis-based lock manager for dispatchChanges on test sites. - https://phabricator.wikimedia.org/T159828#3140724 (10daniel) @hoo Can you confirm that we are actually running multiple instances of dispatchChanges.php for test.wikidata.or... [16:00:49] 10DBA, 10Wikidata, 13Patch-For-Review, 15User-Daniel, and 2 others: Use redis-based lock manager for dispatchChanges on test sites. - https://phabricator.wikimedia.org/T159828#3140726 (10daniel) a:05Ladsgroup>03daniel [16:12:23] 10DBA, 06Operations, 10hardware-requests, 10ops-codfw: codfw: (1) spare pool system for temp allocation as database failover - https://phabricator.wikimedia.org/T161712#3140803 (10RobH) a:05faidon>03RobH [16:41:18] marostegui: it was supposed to happen last week but users requested we postpone it so we did, let us know if you see any problem with us doing it today [16:41:36] nuria: sure, no worries. No problem from our end [16:42:00] marostegui: okeis okeis [16:43:01] marostegui: our plan: https://etherpad.wikimedia.org/p/analytics-el-table-rename [16:43:03] cc ottomata [16:47:51] nuria thanks [16:49:30] marostegui: did you add the thing about terbium [16:49:58] marostegui:"stop backlog filling from terbium to db1047 and dbstore1002?" [16:50:02] cc jynus [16:50:22] nuria: yes, we have to log in on the terbium server [16:50:34] attach to the screens called: 10082.db1047 13861.dbstore1002 [16:50:36] and crtl+c [16:50:51] marostegui: I am not sure what terbium does [16:50:57] we can do that [16:51:24] ottomata: db1047 and dbstore1002 are the oens that need puppet disabled (sorry!) [16:51:48] I will disable it now [16:51:52] oh so it odesn't start up el_sync [16:51:52] k [16:51:59] mark: what's this eventlogging_resync stuff on terbium? [16:52:01] marostegui: ah for replication [16:52:20] oops [16:52:29] that was autocomplete mistakte [16:52:33] marostegui: i meant [16:52:43] yeah, but el_sync runs on 1047 and dbstore1002 [16:52:54] don't know what terbium's eventlogging_resync.sh is [16:53:59] disabling it now [16:54:06] but what is it?! :) [16:54:33] ottomata, remember when the pks were deleted and parallel load was implemented without talkint to me? [16:54:46] that backfills so the slave receives all events [16:55:11] jynus: me no compredou "pks" [16:55:19] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1067 - https://phabricator.wikimedia.org/T161600#3141035 (10Marostegui) 05Open>03Resolved a:03Marostegui Rebuilt ``` root@db1067:~# megacli -PDRbld -ShowProg -PhysDrv [32:11] -aALL ; megacli -ldinfo -l0 -a0 Device(Encl-32 Slot-11) is not in rebuil... [16:55:19] primary keysa [16:55:36] I shouldn't be on this chat! ops [16:56:06] it is now disabled [16:56:45] jynus: let me understand... [16:56:51] does this resync script on terbium have to run in order for EL replication to work? [16:56:53] jynus: replication runs from that machine? [16:56:58] ottomata, yes [16:57:22] there are 2 scripts, the ones on the local machines, and the ones on terbium [16:57:41] the ones on the local machines were puppetized by me [16:58:19] the ones on terbium work when the ones puppetized innevitably fail [16:58:29] and users complain to me [16:58:35] jynus: ahem.. why do they fail? [16:58:52] there's a whole ticket about making that better, so we don't need to revisit tha [16:58:53] ok [16:58:55] because parallel loading performs inserts out of order [16:59:01] jynus: ah ok [16:59:01] jynus: there is not parallel loading [16:59:03] we turned it off [16:59:09] when we realized there were no primary keys [16:59:12] well, inserts hapen out of order [16:59:14] and replication wasn't done with pk [16:59:22] hasn't been on for a year [16:59:26] or more [16:59:39] in any case, those only fill-in the gaps [16:59:44] and they keep inserting [16:59:49] so there is something wrong [17:00:23] things are inserted not in timestamp order, which is what the replication scripts assume [17:00:37] jynus: to sum up: replication is running from two different machines for some reason that i will take me a while to understand and both places need to be on for replication to work, correct? [17:00:37] or the PKs [17:00:47] yes, mostly [17:01:29] there is not replication, only a hack waiting to be better [17:02:12] jynus: ajem, so for today's procedure what matters is that hacky script that we all love (and none of us wrote) is stopped [17:02:23] jynus: from both places in which it is running [17:02:25] both of them, yes [17:02:26] and puppet [17:02:37] marostegui: because puppet will restart it? [17:02:40] yep [17:04:29] ok we are proceeding [17:04:36] el stopepd, stopping el sync and puppet on 'slaves' [17:08:40] marostegui: running scripts on master [17:08:47] \o/ [17:10:21] marostegui: teh tables we dropped ImasgeSomething will fail but everything else should be goof, forgot to remove those [17:10:34] ah good [17:10:36] yea [17:15:22] there is a 2 hour query running on dbstore1002- that may affect the rename process [17:16:19] I think it is not selectin EL, so that should be ok [17:17:43] yeah I checked for long queries on tendril for the log database when they said they'd start [17:18:15] marostegui: ok, el_sync and puppet back on on db1047 and dbstore1002 [17:18:20] you can restart the terbium thing [17:18:25] ok [17:19:08] done [17:22:23] we used to have a check of replication, but this got declined: https://phabricator.wikimedia.org/T124306 [17:26:53] 10DBA, 06Analytics-Kanban, 13Patch-For-Review: Change length of userAgent column on EL tables - https://phabricator.wikimedia.org/T160454#3141118 (10Ottomata) Done! https://etherpad.wikimedia.org/p/analytics-el-table-rename [17:27:08] ok we done! [17:27:46] \o/ [17:28:16] dbstore1002 is now running out of space [17:29:03] the tables that were renamed can be dropped at some point? nuria? [17:29:11] (we can add more G to the lvm in the meantime) [17:31:02] It might be coincidence though that the alert was trigged now? https://grafana-admin.wikimedia.org/dashboard/file/server-board.json?refresh=1m&panelId=17&fullscreen&orgId=1&var-server=dbstore1002&var-network=eth0&from=now-1h&to=now [17:31:26] I thinks there was some inestability [17:31:41] but yeah, in the long run there is not huge bump lately [17:33:12] I gave it some more G [17:34:46] I am going to go and cook some dinner [17:35:13] I hope at least https://en.wikipedia.org/wiki/Up_to_eleven Gs [17:36:15] haha didn't know about that expression! [17:36:26] I need to show you that film [17:36:33] but tomorow [17:36:50] vale! see you tomorrow - thanks for your help :) [17:36:52] as always! [18:25:07] 10DBA, 13Patch-For-Review: run pt-table-checksum on s2 (WAS: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038) - https://phabricator.wikimedia.org/T154485#3141324 (10jcrespo) Only diff: ``` db1036:svwiki Rows are different WHERE old_id BETWEEN 27558001 AND 27559000 ``` [21:06:43] 10DBA, 10Analytics-EventLogging, 06Analytics-Kanban, 13Patch-For-Review: Add autoincrement id to EventLogging MySQL tables. {oryx} - https://phabricator.wikimedia.org/T125135#3142024 (10Ottomata) Just a note: now that EL tables were renamed, and active tables recreated in T160454, they should all have an... [21:16:14] 10DBA, 06Labs, 06Operations: eqiad: (2) hardware access request for labsdb1004 & 5 - https://phabricator.wikimedia.org/T161754#3142042 (10chasemp) [21:16:44] 10DBA, 06Labs, 06Operations: eqiad: (2) hardware access request for labsdb1004 & 5 refresh - https://phabricator.wikimedia.org/T161754#3142056 (10chasemp) [21:18:06] 10DBA, 06Labs, 06Operations: eqiad: (2) hardware access request for labsdb1006 & 7 refresh - https://phabricator.wikimedia.org/T161755#3142057 (10chasemp) [21:55:12] 10DBA, 06Labs, 06Operations: eqiad: (2) hardware access request for labsdb1006 & 7 refresh - https://phabricator.wikimedia.org/T161755#3142279 (10chasemp) a:03RobH