[06:08:34] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Global rename of The_Photographer → Wilfredor: supervision needed - https://phabricator.wikimedia.org/T215107 (10Marostegui) Sorry, I wasn't available during the weekend. I am normally available from Monday to Friday from 7:00 UTC to 16:00 UTC
[06:35:45] <wikibugs>	 10DBA, 10MediaWiki-Database: Convert primary key integers and references thereto from int to bigint (unsigned) - https://phabricator.wikimedia.org/T63111 (10Marostegui) >>! In T63111#4941569, @Krinkle wrote: > Re-opening so as to let the DBAs triage this. One question I wasn't able to answer quickly is: What u...
[07:06:10] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) Now that we know that the biggest and and most painful table (as it was Aria and it was huge - around 180G...
[07:06:19] <marostegui>	 elukey: ^
[07:07:10] <elukey>	 you are awesome
[07:07:15] <elukey>	 thanks a lot
[07:07:24] <elukey>	 I am already asking people to test the new stuff
[07:07:41] <marostegui>	 I am now thinking if we should move staging from dbstore1003 to dbstore1005 to optimize better the disk space
[07:07:46] <marostegui>	 I will do some math and comment on the tas
[07:07:47] <marostegui>	 task
[07:07:59] <marostegui>	 (it is a matter of transferring 130GB, so it shouldn't take long anyways)
[07:08:11] <elukey>	 ack, I'll leave this choice to the expert :)
[07:08:15] <marostegui>	 :)
[07:08:23] <elukey>	 I am going to merge the SRV dns change and complete the code snippets
[07:08:24] <marostegui>	 so that means you!
[07:08:26] <marostegui>	 great
[07:08:28] <elukey>	 yeah sure
[07:09:13] <elukey>	 with the SRV records there is a ton of code that it is not needed in people's script
[07:09:24] <elukey>	 so asking to them to migrate should be easier
[07:09:48] <marostegui>	 awesome :)
[07:10:02] <marostegui>	 I would like to aim for a given day (tbd)
[07:10:09] <marostegui>	 but to kind of set a deadline
[07:14:23] <elukey>	 ;; ANSWER SECTION:
[07:14:23] <elukey>	 _staging-analytics._tcp.eqiad.wmnet. 300 IN SRV	0 1 3340 dbstore1003.eqiad.wmnet.
[07:14:26] <elukey>	 lovely
[07:14:56] <elukey>	 yep yep makes sense, today I am going to send an email to alert people, if you are ok I'd do it next monday
[07:15:17] <marostegui>	 Including the staging migration?
[07:15:22] <elukey>	 or, maybe better, we can ask to migrate staging stuff for say Thursday
[07:15:24] <marostegui>	 (which includes read only)
[07:15:50] <elukey>	 in SF they asked to me to have the two systems in parallel for a bit
[07:16:03] <elukey>	 to have time to re-tune the scripts, test, etc..
[07:16:16] <marostegui>	 but they are aware that changes done to staging on one host will not get replicated to another host, right?
[07:17:20] <elukey>	 not sure, this is why I want to send an email explaining the plan. My idea was to give them a week to prepare for the migration to dbstore1003/5 (depending where we'll put staging) and then do the migration
[07:17:31] <marostegui>	 great
[07:17:42] <marostegui>	 I will check the final destination of staging today
[07:17:51] <elukey>	 ack
[07:18:02] <elukey>	 I am prepping scripts/docs/etc..
[07:18:09] <marostegui>	 What we can do is do a final migration of staging with the latest data (which includes read only time as I specified) one day before or something?
[07:41:20] <elukey>	 (sorry just seen this)
[07:41:29] <elukey>	 yes definitely, as long as everybody is onboard
[07:41:37] <marostegui>	 sure
[07:41:46] <elukey>	 then we set staging to read-only (on dbstore1002)
[07:41:59] <elukey>	 so whoever didn't see the email hopefully will reach out to me
[07:42:03] <marostegui>	 hehe
[07:42:04] <marostegui>	 yes
[07:42:15] <marostegui>	 Keep in mind that we have to set read only to do the staging migration 
[07:42:15] <elukey>	 in the meantime, I added more info to https://wikitech.wikimedia.org/wiki/Analytics/Data_access#MariaDB_replicas
[07:42:23] <elukey>	 even better
[07:42:36] <marostegui>	 oh nice
[08:06:49] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10Patch-For-Review, 10User-Ladsgroup: Drop change_tag.ct_tag column in production - https://phabricator.wikimedia.org/T210713 (10Marostegui)
[08:27:07] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) After having al the sections ready and compressed on all hosts, there is one thought I had, where to leave...
[09:07:31] <marostegui>	 elukey: I am going to move the staging db to dbstore1005 yeah
[09:07:39] <marostegui>	 I will update the SRV records 
[09:08:51] <elukey>	 and also the CNAME
[09:08:54] <elukey>	 but +1
[09:08:55] <marostegui>	 yep
[09:09:00] <marostegui>	 I will add you as a reviewer
[09:24:33] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, and 2 others: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases - https://phabricator.wikimedia.org/T162070 (10jcrespo) > Second, mariadb::packages_wmf and mariadb::packages should probably be merged into one...
[09:43:14] <marostegui>	 elukey: https://gerrit.wikimedia.org/r/#/c/operations/dns/+/489636/
[09:43:44] <wikibugs>	 10DBA: Remove AFT tables from the analytics slaves - https://phabricator.wikimedia.org/T92739 (10Marostegui) 05Open→03Resolved a:03Marostegui The clicktracking no longer exist on either s1 master or dbstore1002: ` root@db1067.eqiad.wmnet[enwiki]> show tables like '%click%'; Empty set (0.00 sec)  root@dbsto...
[09:43:47] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui)
[09:43:50] <wikibugs>	 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui)
[09:45:51] <elukey>	 marostegui: merged and authdns-updated
[09:45:57] <marostegui>	 Thank you!
[10:12:45] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10Patch-For-Review, 10User-Ladsgroup: Drop change_tag.ct_tag column in production - https://phabricator.wikimedia.org/T210713 (10Marostegui)
[10:12:59] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui)
[10:13:46] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui)
[10:14:15] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui)
[10:14:48] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10Patch-For-Review, 10User-Ladsgroup: Drop change_tag.ct_tag column in production - https://phabricator.wikimedia.org/T210713 (10Marostegui) s8 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1005 [] dbstore1002 [] db1124 [] db1116...
[10:17:07] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) >>! In T210478#4942496, @Marostegui wrote: > After having al the sections ready and compressed on all host...
[10:42:52] <jynus>	 I have uploaded 10.1.38-MariaDB, seems to work ok
[10:43:20] <marostegui>	 \o/
[10:44:08] <jynus>	 should I add db1114 to tendril?
[10:44:18] <marostegui>	 db1114 should already be there no?
[10:44:47] <jynus>	 it had to be removed to take away the topology while it was down
[10:44:52] <marostegui>	 aaaaah
[10:44:53] <marostegui>	 right
[10:44:54] <marostegui>	 sure
[10:45:45] <marostegui>	 we don't have many core hosts  not running 10.1.37 in eqiad anymore
[10:45:51] <marostegui>	 only a few on s8 I think
[10:49:49] <jynus>	 dumps finised on db1106, so we can restart that one now
[10:50:00] <jynus>	 once I pool db1118
[12:18:34] <marostegui>	 jynus: can you paste the cve or mariadb link on the sre etherpad? I cannot find it now
[12:19:06] <jynus>	 let me see
[12:20:20] <jynus>	 some of them are not public, I would redirect to something like https://www.percona.com/blog/2019/02/06/percona-responds-to-mysql-local-infile-security-issues/
[12:21:23] <jynus>	 as it provides an easy to read config or upgrade suggetsions
[12:58:13] <wikibugs>	 10DBA, 10Analytics, 10Research: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10diego)
[13:02:28] <wikibugs>	 10DBA, 10Analytics, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10diego)
[14:06:46] <marostegui>	 thanks!
[14:11:10] <wikibugs>	 10DBA, 10Analytics, 10MediaWiki-Database, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10Marostegui)
[16:07:21] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Marostegui: rack/setup/install db11[26-38].eqiad.wmnet - https://phabricator.wikimedia.org/T211613 (10Marostegui) @Cmjohnson I can take care of the installations once you've done the RAID and added DNS and pxeboot entries with the MACs :-)
[16:35:19] <wikibugs>	 10DBA, 10MediaWiki-Database, 10Core Platform Team Backlog (Watching / External), 10Performance-Team (Radar), 10Wikimedia-Incident: Fix mediawiki heartbeat model, change pt-heartbeat model to not use super-user, avoid SPOF and switch automatically to the real mast... - https://phabricator.wikimedia.org/T172497
[16:41:27] <wikibugs>	 10DBA, 10Patch-For-Review: Drop valid_tag table - https://phabricator.wikimedia.org/T212254 (10GTirloni)
[16:43:38] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) a:05elukey→03Marostegui
[16:44:38] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui)
[16:45:51] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui)
[16:46:59] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui)
[16:47:41] <wikibugs>	 10DBA, 10Analytics, 10MediaWiki-Database, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10Nuria) p:05Triage→03High
[16:56:15] <jynus>	 db1106 of course stuck on loading ramdisk
[16:56:39] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui)
[16:56:46] <marostegui>	 jynus: same case as db2085??
[16:57:06] <jynus>	 probably, it rebooted on its own
[16:57:36] <jynus>	 not booting a second time
[16:57:44] <jynus>	 should I force the older kernel?
[16:57:45] <marostegui>	 :(
[16:57:49] <marostegui>	 yeah
[16:58:04] <marostegui>	 I think moritzm mentioned on the task trying to enable debugging on intrd
[16:58:24] <jynus>	 no time for that now
[16:58:32] <jynus>	 could you note it on the task, cannot find it now
[16:59:04] <jynus>	 yeah, it enter an infinite reboot loop
[16:59:22] <marostegui>	 I am not saying to do it _now_ I mean as a next step
[16:59:26] <jynus>	 PowerEdge R630
[17:01:25] <wikibugs>	 10DBA, 10Operations, 10Packaging: db2085 doesn't boot with 4.9.0-8-amd64 - https://phabricator.wikimedia.org/T214840 (10Marostegui) Same thing just happened with db1106 (PowerEdge R630 - same chassis as db2085) @MoritzMuehlenhoff can you help us with the approach you mentioned at T214840#4918369 ?
[17:02:00] <jynus>	 thanks
[17:03:04] <wikibugs>	 10DBA, 10Analytics, 10MediaWiki-Database, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10JAllemandou) @diego : This has worked for me (takes some time to compute and needs a bunch of resources). I hope it's close enough...
[17:03:34] <jynus>	 it doesn't boot on -6 either
[17:03:38] <jynus>	 :-(
[17:04:01] <marostegui>	 :|
[17:04:02] <jynus>	 I don't have any other kernel available
[17:04:10] <marostegui>	 that is very strange
[17:04:22] <marostegui>	 any error logged? (hw error I mean)
[17:04:26] <marostegui>	 on the idrac
[17:06:40] <jynus>	 it finally booted with -6
[17:06:43] <jynus>	 :-P
[17:07:08] <marostegui>	 pheeeeew
[17:07:15] <jynus>	 please mention this issue on the codfw restart
[17:07:24] <wikibugs>	 10DBA, 10Operations, 10Packaging: db2085/db1106 don't boot with 4.9.0-8-amd64 - https://phabricator.wikimedia.org/T214840 (10Marostegui)
[17:07:25] <marostegui>	 codfw restart?
[17:07:35] <jynus>	 the "rolling restart"
[17:07:44] <marostegui>	 ah yes
[17:07:47] <marostegui>	 I added it as a line
[17:07:49] <marostegui>	 earlier today
[17:07:50] <marostegui>	 line 199
[17:11:20] <jynus>	 as in, hey, this is happening, be careful when restarting
[17:11:32] <jynus>	 not need for help (except moritz) for now
[17:13:43] <moritzm>	 are db2085 and db1106 from the same hw batch/class?
[17:13:52] <marostegui>	 same HW yes
[17:13:56] <marostegui>	 same batch, don't know
[17:14:28] <moritzm>	 seems so, yes. very similar purchase date
[17:16:12] <moritzm>	 we can narrow this down further by testing the 4.9.144-3 kernel from the stretch point update (pending for next weekend, but we can already test the new kernel), I need to leave after the SRE meeting, but happy to investigate further tomorrow morning
[17:17:47] <wikibugs>	 10DBA, 10Operations, 10Packaging: db2085/db1106 don't boot with 4.9.0-8-amd64 - https://phabricator.wikimedia.org/T214840 (10Marostegui)
[17:18:06] <marostegui>	 thanks moritzm
[17:23:27] <jynus>	 moritzm: sorry for the ping
[17:23:43] <jynus>	 I was just getting worried because meeting + that host being more difficult to failover
[17:23:57] <jynus>	 we wasted the restart because we wen't back to -6
[17:24:03] <jynus>	 *went
[17:27:34] <moritzm>	 ack, np wrt ping, happy to help/fix this
[17:29:00] <jynus>	 oh, we will try to look at this ourselves, don't worry, but obviously guidance will be helpful
[17:46:41] <wikibugs>	 10DBA, 10Operations, 10Packaging: db2085/db1106 don't boot with 4.9.0-8-amd64 - https://phabricator.wikimedia.org/T214840 (10Marostegui) @paravoid gave us some food for thought: ` stuck at "loading ramdisk" is sometimes an indication of misconfigured serial redirection after boot basically when Linux and the...
[18:23:43] <wikibugs>	 10DBA, 10Analytics, 10MediaWiki-Database, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10diego) Looks good @JAllemandou, thanks. This is a good workaround, but imho,  we should have an structure or schema that makes this...
[18:28:05] <wikibugs>	 10DBA, 10Analytics, 10MediaWiki-Database, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10EBernhardson) I don't know if this meets your needs, but the cirrussearch dumps have the wikidata id's broken out.  This is the `wi...
[19:49:35] <jynus>	 db1083 looks like it is slow than average to connect, maybe we can do better connection rebalancing weights, someting to look at tomorrow
[19:50:19] <jynus>	 apparently it was just a spike, so nm
[19:51:25] <wikibugs>	 10DBA, 10Analytics, 10MediaWiki-Database, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10diego) @EBernhardson , this looks exactly what I was looking for, initially.  Thank you very much for that.  However, I wont close...
[19:53:13] <wikibugs>	 10DBA, 10Analytics, 10MediaWiki-Database, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10EBernhardson) >>! In T215616#4944986, @diego wrote: > @EBernhardson , this looks exactly what I was looking for, initially.  Thank...
[20:07:29] <wikibugs>	 10DBA, 10Analytics, 10MediaWiki-Database, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10jcrespo) > diego added a project: DBA.  I don't understand what is the actionable here for us. Without context, I would say that:...
[22:34:49] <wikibugs>	 10DBA, 10Analytics, 10MediaWiki-Database, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10diego) @jcrespo, the API works good for query specific pages/entities, not for example to know which pages that existing in X_wiki...