[00:47:21] 10DBA, 06Community-Tech, 10MediaWiki-User-blocking: Do test queries for range contributions to gauge performance of using different tables - https://phabricator.wikimedia.org/T156318#3123970 (10MusikAnimal) I am in the process of getting prod db access and will get back to you. Rest assured I won't forget ab... [03:55:14] 10DBA, 06Labs, 10Tool-Labs, 07Regression: Tool Labs: Add skin, language, and variant to user_properties_anon - https://phabricator.wikimedia.org/T152043#2836353 (10bd808) Does anybody know how to accomplish this change to the user_properties_anon view? [07:23:13] 10DBA: Cannot access the database: Can't connect to MySQL server on '10.192.48.41' (111) (10.192.48.41) - https://phabricator.wikimedia.org/T161159#3124243 (10Marostegui) I believe es2016 is ready to come back up as the transfer to es2015 is finished (or looks so from the screen that is in both hosts). I will wa... [07:32:17] 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: *_minor_mime are varbinary(32) on WMF sites, out of sync with varbinary(100) in MW core - https://phabricator.wikimedia.org/T73563#3124246 (10Marostegui) db1068 and labsdb1011 are done: ``` root@neodymium:~# for i in labsdb1011.eqiad.wmnet db1068.eqiad.... [07:33:05] 07Blocked-on-schema-change, 10DBA, 06Multimedia, 05MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), and 3 others: Review schema changes for T125071 - Add index to image table on all wikis - https://phabricator.wikimedia.org/T160415#3124247 (10Marostegui) db1068 and labsdb1011 are done: ``` root@neo... [07:39:02] 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: *_minor_mime are varbinary(32) on WMF sites, out of sync with varbinary(100) in MW core - https://phabricator.wikimedia.org/T73563#3124254 (10Marostegui) >>! In T73563#3124253, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operatio... [07:39:12] 07Blocked-on-schema-change, 10DBA, 06Multimedia, 05MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), and 3 others: Review schema changes for T125071 - Add index to image table on all wikis - https://phabricator.wikimedia.org/T160415#3124255 (10Marostegui) >>! In T160415#3124252, @Stashbot wrote: > {... [08:03:46] 10DBA: Cannot access the database: Can't connect to MySQL server on '10.192.48.41' (111) (10.192.48.41) - https://phabricator.wikimedia.org/T161159#3124260 (10jcrespo) I am checking the transfer was done correctly checksumming the files, I will put up the servers soon. [08:58:47] 10DBA: Cannot access the database: Can't connect to MySQL server on '10.192.48.41' (111) (10.192.48.41) - https://phabricator.wikimedia.org/T161159#3124295 (10jcrespo) 05Open>03Resolved a:03jcrespo I am sorry this happened, this was a calculated risk. I needed to clone from the master, but depooling is imp... [09:07:28] 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3124309 (10jcrespo) > We (WMDE) will discuss the readonly side of things in our meeting on Tuesday What is the res... [09:21:21] do you think we will have enough disk space for the alter on labsdb1001? [09:24:43] The table was around 150G on production [09:24:49] so I assumed it will be less in labsdb [09:24:59] And we had around 380G free [09:25:01] in /srv/ [09:25:03] so we will see [09:26:27] it is not failing, which is in itself a win [09:26:40] it failed for me on enwiki-page [09:27:04] why did it fail for you? metadata? [09:27:11] no [09:27:17] metadata prevented it to start [09:27:29] but I eventualy could with some...methods [09:27:54] but it failed in the middle, I assume because traffic in writes or reads (buffering) [09:28:02] ah right [09:28:24] we will see it if fails once the traffic ramps up a bit [09:28:54] I see es2015 went fine [09:29:02] I thought it would take longer, given the SATA disks [09:29:05] I had to use the master [09:29:09] Ah yes [09:29:13] I had a question about that [09:29:19] Why are some of them compressed? [09:29:43] I tested before the first failover [09:29:57] itr turns out gzip content doesn't gzip very well [09:30:27] I am still waiting for the time to undo that [09:30:33] which probably it is today [09:30:48] So the impact you saw what was? [09:30:55] impact? [09:31:12] "˜/jynus 10:29> itr turns out gzip content doesn't gzip very well " [09:31:30] So, you didn't like the compressed tables, what did you see? [09:31:33] I wasted a day but got no dfisk gains [09:31:39] ah right [09:31:41] *disk [09:31:44] got it [09:31:58] this is still when I was testing compression [09:32:12] and I was excited about the gains on regular dbs [09:32:14] so you want to go back to non compressed tables? [09:32:25] then I tested it on this es server [09:32:33] and it had no gains [09:32:44] becase mediawiki already stores compressed data [09:32:59] but did it had issues? so, is there any gain going back to non compressed tables? [09:33:07] no [09:33:25] no issue, no gain, no loss [09:33:35] it is just a waste of time [09:33:47] that is why I never had the time to put it back on its place [09:34:00] on its original format [09:34:02] so, we'll leave es2014 as it is now, compressed? or you want to reclone it? [09:34:06] no [09:34:12] I want to uncompress it [09:34:33] it is easier becase it only involves 1 server [09:34:52] so it is just 800 alters, one per db [09:35:02] haha "just" [09:35:09] well, it could be worse [09:35:10] is that faster than a reclone? [09:35:19] it is 40.000 on a regular table [09:35:23] it is safer [09:35:28] less servers involved [09:35:33] can be done fully online [09:35:46] and we do not clone everyhing from the master [09:36:16] which was my only dissapointment with losing es2015 [09:37:50] not even all are compresseed [09:37:55] so much faster [09:37:59] enwiki is not compressed [09:38:30] ah I see I see [09:38:41] *I am learning about this now [09:38:46] as I look at it [09:40:07] sure sure, I just asked to know a bit the history of it [09:40:10] I think only the smallest 907 tables are compressed [09:40:15] it was a test [09:40:21] It went wrong [09:40:35] or good in that compression was not helpful there [09:40:42] at least we know :) [09:40:49] that what the tests are for! [09:41:06] data does not have any problem [09:41:14] but the format is useless [09:41:26] I will start a screen comverting it back [09:41:33] *converting it back [09:41:42] ok [09:41:43] and then retire the comment [09:44:23] 10DBA, 13Patch-For-Review: Defragment db1070, db1082, db1087, db1092 - https://phabricator.wikimedia.org/T137191#3124358 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db1082.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimag... [09:48:23] 10DBA, 06Operations, 10ops-codfw: Several es20XX servers keep crashing (es2017, es2019, es2015, es2014) since 23 March - https://phabricator.wikimedia.org/T130702#3124380 (10jcrespo) [09:48:26] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: es2015 crashed on 2017-03-11 - https://phabricator.wikimedia.org/T160242#3124379 (10jcrespo) 05Open>03Resolved [09:50:44] 10DBA, 06Operations, 10ops-codfw: Several es20XX servers keep crashing (es2017, es2019, es2015, es2014) since 23 March - https://phabricator.wikimedia.org/T130702#3124383 (10jcrespo) es2015 crashed on 2017-03-11, faulty cpu and board replaced. [09:56:48] 10DBA, 13Patch-For-Review: es2014 revert data compression - https://phabricator.wikimedia.org/T129350#3124430 (10jcrespo) a:03jcrespo [10:24:47] 10DBA, 13Patch-For-Review: Defragment db1070, db1082, db1087, db1092 - https://phabricator.wikimedia.org/T137191#3124506 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db1082.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimag... [10:37:49] 10DBA, 13Patch-For-Review: Defragment db1070, db1082, db1087, db1092 - https://phabricator.wikimedia.org/T137191#3124516 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db1082.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimag... [10:48:37] FYI https://github.com/prometheus/mysqld_exporter/pull/191/files#diff-0 [10:49:58] heartbeat is not enough for us [10:50:20] and I will shoot by saying: how do you handle prometheus-exported changing needs? [10:50:26] *exporter [10:51:00] do we need to recompile every time we need to change it aside from the enable-disable full modules? [10:51:42] because we can monitor mysql with the deafault exporters, no problems [10:51:58] but we cannot monitor wikimedia-databases with them [10:52:27] the changes are mostly trivial, but they could be constant and frequent [10:59:18] 10DBA, 13Patch-For-Review: Defragment db1070, db1082, db1087, db1092 - https://phabricator.wikimedia.org/T137191#3124546 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1082.eqiad.wmnet'] ``` and were **ALL** successful. [11:11:40] jynus: what kind of changes do you have in mind? [11:12:12] godog, in a meeting let's talk later [11:12:15] ok! [11:12:28] ping me here otherwise I'll miss it [12:09:26] 07Blocked-on-schema-change, 10DBA, 06Multimedia, 05MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), and 3 others: Review schema changes for T125071 - Add index to image table on all wikis - https://phabricator.wikimedia.org/T160415#3124770 (10Marostegui) db1059 and labsdb1001 are done: ``` root@neo... [12:10:45] 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: *_minor_mime are varbinary(32) on WMF sites, out of sync with varbinary(100) in MW core - https://phabricator.wikimedia.org/T73563#3124774 (10Marostegui) db1059 and labsdb1001 are done: ``` root@neodymium:~# mysql --skip-ssl -hdb1059 commonswiki -e "show... [12:41:53] 10DBA, 06Operations, 05DC-Switchover-Prep-Q3-2016-17, 07Wikimedia-Multiple-active-datacenters: Decouple Mariadb semi-sync replication from $::mw_primary - https://phabricator.wikimedia.org/T161007#3124830 (10jcrespo) a:03jcrespo [14:38:53] 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3125233 (10Lea_WMDE) >>! In T130067#3124309, @jcrespo wrote: >> We (WMDE) will discuss the readonly side of things... [15:38:33] jynus: Hi, I rolled out changes on T158420 yesterday and now both tool accounts and tools project users get labsdbaccounts created on the go. The labsdbaccounts meta database in m5-master has these users set as type user in the account table. [15:38:33] On the new labsdbs boxes, I used the same labsdbuser role that we've been applying for tool accounts - chasemp and I were taking this morning and wondering if we should have used a different role name for these accounts and if it would matter to you administratively. [15:38:34] T158420: Make maintain-dbusers.py create replica.my.cnf files for user accounts as well - https://phabricator.wikimedia.org/T158420 [15:38:55] talking* [15:39:38] madhuvishy, send a proposal- it takes very little time to change it [15:39:49] do you see a need for it? [15:39:52] meaning [15:40:13] will users and tools have different grants in the future? [15:40:24] jynus: mostly wondering if it matters it's different if we wanted to revoke a role (dba admin purposes) [15:40:30] our thinking was we may want to have different rules for users vs tools as users shouldn't be running any long running services [15:40:32] not for us [15:40:38] ah [15:40:40] and grants question is open, maybe for user db's? [15:40:49] do users have a different name? [15:40:51] re: grant divide [15:40:55] yes [15:41:02] yeah they match u% [15:41:02] s1234 vs u1234 [15:41:04] then I think different roles is not needed [15:41:08] cool [15:41:10] and exist in meta detabase [15:41:11] we can add it in the future [15:41:14] if we need it [15:41:18] database* [15:41:19] for queries [15:41:20] cool [15:41:26] we can filter by user name [15:41:40] e.g. prevent long running queries for u% [15:41:48] so yes, I think it is cool [15:41:54] my only petition [15:42:07] is to put all of that in writing somewhere on wikitech [15:42:16] a single line is enough [15:42:22] sure will do [15:42:22] no need for a full manual [15:42:42] "u%" as users X and also have acces Y [15:42:45] *are [15:43:04] I still need to document thing myself [15:43:14] regarding grants [15:43:15] cool [15:43:26] thanks for working on that, madhuvishy and chasemp [15:43:27] ! [15:43:34] one note - in the old labsdbs there are about 6k users matching u% [15:43:42] madhuvishy: did the magic [15:43:49] thanks, madhuvishy [15:44:02] the new ones i only created now, and there are only 1027 users who are actually tools users [15:44:03] and chasemp even if you do not want to take credit, too [15:44:21] madhuvishy, that is cool to me [15:44:26] and probably better [15:44:39] agreed [15:44:40] do you think there will be users complaining about lack of access? [15:45:00] 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: *_minor_mime are varbinary(32) on WMF sites, out of sync with varbinary(100) in MW core - https://phabricator.wikimedia.org/T73563#3125407 (10Marostegui) labsdb1003 is done: ``` [root@labsdb1003 15:44 /root] # mysql --skip-ssl commonswiki -e "show create... [15:45:05] https://www.irccloud.com/pastebin/v4wIpAzf/ [15:45:08] or those will be on a ask-serve bases? [15:45:19] no, I've generally said you wnt to use your personal tool labs /home mysql creds in your own project that's ok though for now [15:45:24] since we don't have a viable alt [15:46:02] you are the bosses there [15:46:05] I mean that [15:46:10] right [15:46:13] 07Blocked-on-schema-change, 10DBA, 06Multimedia, 05MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), and 3 others: Review schema changes for T125071 - Add index to image table on all wikis - https://phabricator.wikimedia.org/T160415#3125408 (10Marostegui) labsdb1003 is done: ``` [root@labsdb1003 15... [15:46:15] as in, account handling, we help with mysql [15:46:26] but you can freely take those decisions [15:46:46] ask if you need optinion or help, and we will help, ofcourse [15:46:57] :) [15:47:00] I think we get a lot in sync [15:47:07] with the goal last year [15:47:14] jynus you are the pope of db operations, we are lowely, lowely friars [15:47:15] ha ha [15:47:32] :D [15:47:33] cool then, I shall document current status, and leave things as they are :) [15:47:38] thanks jynus [15:47:40] == I tell other people what to do but do nothing [15:47:43] :-) [15:47:45] heh [15:48:35] The only issue in the past was the \_\_ [15:48:44] and you did handle it marvelously [15:53:53] 10DBA, 06Labs, 06Operations, 07Tracking: Database replication problems - production and labs (tracking) - https://phabricator.wikimedia.org/T50930#3125426 (10Dispenser) [16:00:15] 10DBA, 06Labs, 06Operations, 07Tracking: Database replication problems - production and labs (tracking) - https://phabricator.wikimedia.org/T50930#3125440 (10jcrespo) [18:05:36] 10DBA, 10MediaWiki-Database, 07User-notice: Optimize SpecialAllPages::showChunk for large wikis - https://phabricator.wikimedia.org/T160983#3125789 (10Trizek-WMF) [18:18:04] 10DBA, 10MediaWiki-Database, 07Easy, 07Schema-change: core - archive database table - database schema is inconsistent with update patch - https://phabricator.wikimedia.org/T104756#1426222 (10Reedy) My dev wiki only has the old non prefixed one (and it's been around quite a few versions!) ```lang=sql mysql... [18:31:08] 10DBA, 10MediaWiki-Database: Should we bump minimum supported MySQL Version? - https://phabricator.wikimedia.org/T161232#3125897 (10Reedy) [18:32:06] 10DBA, 10MediaWiki-Database, 07Easy, 13Patch-For-Review, 07Schema-change: core - archive database table - database schema is inconsistent with update patch - https://phabricator.wikimedia.org/T104756#3125923 (10Reedy) >>! In T104756#1428651, @jcrespo wrote: > I will investigate what is the status for WMF... [18:48:38] 10DBA, 10Wikidata, 07Performance, 15User-Daniel, and 2 others: Build an environment to test change dispatching using Redis-based locking - https://phabricator.wikimedia.org/T155190#3125982 (10Ladsgroup) [19:02:47] 10DBA, 06Operations, 05DC-Switchover-Prep-Q3-2016-17, 13Patch-For-Review, 07Wikimedia-Multiple-active-datacenters: Decouple Mariadb semi-sync replication from $::mw_primary - https://phabricator.wikimedia.org/T161007#3126018 (10Volans) @jcrespo if I understand the patch correctly this means that we'll ac... [20:42:40] 10DBA, 10Wikidata, 07Performance, 15User-Daniel: DispatchChanges: Avoid long-lasting connections to the master DB - https://phabricator.wikimedia.org/T151681#3126390 (10daniel) [20:42:42] 10DBA, 10Wikidata, 13Patch-For-Review, 07Performance, and 3 others: Implement ChangeDispatchCoordinator based on RedisLockManager - https://phabricator.wikimedia.org/T151993#3126388 (10daniel) 05Resolved>03Open Reopening until this is tested and confirmed. [21:14:29] 10DBA, 10MediaWiki-Database: Update schema for wmf wiki's for the table archive, migrating to new index ar_usertext_timestamp - https://phabricator.wikimedia.org/T161252#3126591 (10Paladox) [21:15:31] 10DBA, 10MediaWiki-Database: Update schema for wmf wiki's for the table archive, migrating to new index ar_usertext_timestamp - https://phabricator.wikimedia.org/T161252#3126608 (10Paladox) p:05Triage>03High Changing status to reflect this needs to be done as soon as possible. This is also blocking fixing... [21:48:09] 10DBA, 10MediaWiki-Database: Update schema for wmf wiki's for the table archive, migrating to new index ar_usertext_timestamp - https://phabricator.wikimedia.org/T161252#3126810 (10Reedy)