[00:22:10] DBA, Community-Tech, MediaWiki-extensions-PageAssessments: Create tables for PageAssessments in enwiki database - https://phabricator.wikimedia.org/T139552#2435765 (kaldari) [00:54:51] DBA, Community-Tech, MediaWiki-extensions-PageAssessments: Create tables for PageAssessments in enwiki database - https://phabricator.wikimedia.org/T139552#2435821 (kaldari) [07:10:51] DBA, Community-Tech, MediaWiki-extensions-PageAssessments: Create tables for PageAssessments in enwiki database - https://phabricator.wikimedia.org/T139552#2436286 (jcrespo) a:jcrespo>kaldari I searched the table names on mediawiki.org and found not matches. Because of the huge mess on enwiki... [07:31:54] DBA, Community-Tech, MediaWiki-extensions-PageAssessments: Create tables for PageAssessments in enwiki database - https://phabricator.wikimedia.org/T139552#2436313 (kaldari) @jcrespo: Is something like [[ https://www.mediawiki.org/wiki/Extension:PageAssessments#Database_tables | this ]] what you have... [07:36:19] DBA, Community-Tech, MediaWiki-extensions-PageAssessments: Create tables for PageAssessments in enwiki database - https://phabricator.wikimedia.org/T139552#2436328 (jcrespo) a:kaldari>jcrespo Yes, thank you. [09:05:33] DBA, Community-Tech, MediaWiki-extensions-PageAssessments: Create tables for PageAssessments in enwiki database - https://phabricator.wikimedia.org/T139552#2435765 (jcrespo) p:Triage>Normal [12:17:10] DBA, Community-Tech, MediaWiki-extensions-PageAssessments, Schema-change: Create tables for PageAssessments in enwiki database - https://phabricator.wikimedia.org/T139552#2437070 (Danny_B) [12:33:12] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2437164 (Peachey88) [12:35:56] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2437174 (Peachey88) >>! In T54921#2432024, @Krenair wrote: > Also, so that we're keeping track, these are the entries on @jcrespo's list that I don't think anyone... [12:44:07] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2437184 (jcrespo) That is a great idea, but probably we should migrate it to mw.org ? [12:46:37] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2437189 (Reedy) [15:16:25] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2437666 (Reedy) [15:18:02] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#550765 (Reedy) [15:21:50] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2437675 (Reedy) [15:22:58] jynus: I'm presuming you're not wanting to remove all empty/seemingly unused tables [15:23:05] Just know what they're for, why they're there etc? [15:23:05] DBA, Labs, Tool-Labs, Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2437691 (jcrespo) [15:23:14] and hopefully, if they can be removed, remove them [15:23:14] DBA, Labs, Tool-Labs, Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2437677 (yuvipanda) @Magnus since I think he wrote petscan? [15:23:20] Reedy, no, I want to remove them all [15:23:32] what is the difference? [15:23:39] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#550765 (Reedy) [15:23:44] Well, some tables are there in MW core... [15:23:51] But we don't use them anymore due to moving them to other things.. [15:23:54] like job queue to redis [15:24:01] That's not to say MW won't try and use the table [15:24:13] in that case [15:24:17] we will have the tables [15:24:18] but they are still current, active, and in use for other people [15:24:20] but empty [15:24:25] on our infra [15:24:31] I bet they still have data [15:24:37] heh [15:24:40] some of them might [15:24:47] it is a data leak risk [15:24:51] and management overhead [15:25:03] but please! let's move to a wiki [15:25:14] editing a phab ticket is a curse [15:25:21] in fact [15:25:26] heh [15:25:30] if they had to stay for any reason [15:25:36] but it was clearly docuemented [15:25:42] I would be also ok [15:25:46] Makes sense [15:25:56] As long as we keep said documentation somewhere "sensible" [15:26:06] it is not deletion for deletion's sake [15:26:15] it has caused operational problem (outages) already [15:26:22] funny exameple: [15:26:23] yeah [15:26:27] there is a memory table [15:26:34] that is partially deleted [15:26:40] but if they're ancient, unused, and large... getting rid really makes sense [15:26:51] you say, no problem, if it is unused, no problem right? [15:27:06] well, some of the tables... [15:27:08] do you know what memory tables do on server start? [15:27:14] MW hasn't been accessing them for ages [15:27:19] perform a TRUNCATE TABLE for replication [15:27:29] at "best", we still have a maintenance script to migrate it [15:27:32] which may be broken [15:27:33] which , on replication, for some slaves [15:27:38] break replication [15:27:45] and puts all wikis in read only mode [15:27:53] only for a "harmless" table [15:28:14] so, empty tables are ok, if they are on code [15:28:54] but I would say 60% of those come from old code now not deployed anymore [15:29:00] core changes [15:29:11] or uninstalled extensions [15:29:16] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2437701 (Reedy) [15:29:28] Yeah [15:29:44] There's some extensions, like SecurePoll, that aren't used on a regular basis [15:29:48] and above all, the largest ones [15:30:03] take up to 30 GB [15:30:31] I did some math and that was around 2TB of SSDs in total [15:30:36] as it is replicated many times [15:30:52] It's crazy :) [15:31:01] we do not need to save space [15:31:38] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2437708 (Reedy) [15:31:40] but technical debt will make our job more and more complex [15:32:31] Where should it live on wiki? [15:32:34] mw? wikitech? [15:32:46] one of both [15:32:53] technically it should be wikitech [15:32:58] but I prefer mediawiki [15:33:13] as it would be closer to hackers [15:33:42] We really don't have a clear definition as to what should live where [15:34:37] I supposed mediawiki is for mediawiki and wikitech is for wikimedia-tech [15:34:47] it's what it's supposed to be [15:34:47] but of course there is no line there [15:34:51] But we get stuff going either way [15:34:55] yes [15:35:01] I popose mediawiki [15:35:10] to have in the future easy cross-documentation [15:35:16] look at what I asked here: [15:35:32] https://phabricator.wikimedia.org/T139552#2436286 [15:36:07] We could have a more complete https://www.mediawiki.org/wiki/Manual:Database_layout#Version_history [15:36:21] and I volunteer to do that [15:36:31] with tables that actually exist [15:36:39] heh [15:36:39] and a link to where they came [15:37:02] I can do that, and that table is a good starting point [15:37:12] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2437738 (Reedy) [15:37:38] when you finish editing, please ping me and I will migrate it somewhere [15:37:59] there are some tables [15:38:03] that are certainly needed [15:38:13] for operational reasons [15:38:21] like blobs_tracking [15:38:24] and similar [15:38:29] but they should not be on the cluster [15:38:52] because we do not need 20 copies of it! [15:39:15] I can create a special place on a different cluster, with only 2 or 3 copies [15:39:42] yeah [15:39:42] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2437757 (Reedy) [15:39:48] I guess, it's mostly people not cleaning up after themselves [15:40:42] stuff like that that is explicitly transient [15:41:15] but it is ok if it was not on our "most expensive" cluster [15:41:50] and here expensive not only means disks or servers, also time spent mantaining it and risk it has to break replication [15:41:57] also there are other factors [15:42:05] There's quite a few tables that you can start dropping [15:42:06] m3 may have like 40000 tables [15:42:13] *s3, I mena [15:42:25] if we drop 10 tables per wiki [15:42:35] that is 9000 less tables [15:42:59] right now upgrading/starting/stopping m3 servers take minutes-hours [15:43:18] because for some tasks it has to individually check every table [15:43:38] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2437760 (Reedy) [15:43:40] it has reached a moment in which the clean up is long-time needed! [15:44:14] We've been filing bugs for a while ;) [15:46:16] and I've been deleting tables for a while, too [15:46:40] -rw-rw---- 1 mysql mysql 1048576 Aug 28 2014 job.ibd [15:46:48] Does 1048576 mean empty? [15:46:52] (effectively) [15:46:58] I can check [15:46:58] or, it's just minimum disk space allocation [15:47:12] there is a minimum extent size [15:47:30] like I say, the job table should be empty on every wiki [15:47:31] aslo there could be non-useful data [15:47:48] think that .ibds do not shrink in size [15:48:03] so unles defragmented, they always have the largest size they got to [15:48:25] There's quite a few tables with that size figure :) [15:48:28] and technically, they are not defragmented, only dropped and recreated [15:51:31] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2437784 (jcrespo) ``` MariaDB PRODUCTION s1 localhost enwiki > SELECT table_name FROM information_schema.tables where table_schema='enwiki' and table_rows=0; +----... [15:51:40] https://phabricator.wikimedia.org/T54921#2437784 [15:51:56] ah, thank you wikibugs for coping it before me! [15:52:16] lol [15:55:03] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2437795 (Reedy) [15:55:36] I created __wmf_checksums [15:56:03] I prefixed with __wmf_ to note it was not a mediawiki table, but a wmf-operations required one [15:56:14] we have some of those, such as heartbeat.heartbeat [15:56:16] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#550765 (Reedy) [15:56:55] A quick glance at the list, there should be at least 10 extension tables that can easily go, due to being undeployed for a while [15:57:07] I didn't create _image_new, but I assume it is a failed pt-online-schema-change of the image table, and can go [15:57:22] (we will probably not find it on other servers) [15:57:28] yeah [15:57:36] we're nothing if not inconsistent [15:57:45] it is ok [15:57:57] if we have the other tables documented and identified [15:58:07] the ex prefixed tables seem to be backups before schema changes etc [15:58:48] yes [15:58:59] for those we should not document them [15:59:05] but standarize a procedure [15:59:27] yeah [15:59:28] so that we immediately know "oh, those are leftovers from that emergency/backup/etc." [15:59:34] no provedure [15:59:38] ^ex, ^old, _old$ [15:59:39] just a name we all use [15:59:44] not 50 [16:00:03] maybe tablename_T1234 [16:00:45] tables also have space for comments, but I have never seen anyone using them [16:00:47] DBA, Labs, Tool-Labs, Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2437810 (Magnus) The five threads are for bot accounts only. Normal user accounts get single thread with delay. I have used my own bot account //a lot// over the years, with previo... [16:01:25] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2437811 (Reedy) [16:01:43] I wonder if we should split the tables... [16:01:52] Ones that definitely should not be removed, should be moved [16:07:24] DBA, Labs, Tool-Labs, Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2437819 (jcrespo) @Magnus, as you can see on the discussion I agreed with you initially, and in no way I am giving you any responsibility for this particular incident. However, the... [16:11:32] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2437836 (jcrespo) Are you sure they are still running?- I commented them on puppet and commented it from the server. See: ``` # HEADER: This file was autogenerated at 2... [16:14:08] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2437858 (demon) They were still running on phab2001, which was causing cronspam that @faidon alerted me to this morning. [16:16:35] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2437887 (jcrespo) phab2001 connects to m3-slave? That is even a worse problem! Are you using TLS?- the answer is no, because until now it did not work due to 5.5) [16:20:36] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2437907 (demon) >>! In T138460#2437887, @jcrespo wrote: > phab2001 connects to m3-slave? That is even a worse problem! Are you using TLS?- the answer is no, because unti... [16:26:05] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2437926 (jcrespo) @demon I agree (you will get mails twice). Not a huge issue because db1048 is mostly up, only depooled still because I found some data differences with... [16:35:08] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2437957 (jcrespo) Please continue working on phab architecture. There is already a slave on codfw: db2012 [17:12:44] DBA, Operations, ops-codfw: pc2006 down - https://phabricator.wikimedia.org/T139283#2438116 (Papaul) Bios update from 1.5.4 to 2.1.7 [17:18:36] DBA, Labs, Tool-Labs, Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438159 (Magnus) Just run it as a normal user and not a bot user! [17:52:45] DBA, Labs, Tool-Labs, Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438321 (jcrespo) I will see what the user responds, and act depending on it. [17:54:08] DBA, Operations, ops-codfw: pc2006 down - https://phabricator.wikimedia.org/T139283#2438327 (jcrespo) ``` /admin1-> racadm getsel Record: 1 Date/Time: 12/18/2015 20:35:18 Source: system Severity: Ok Description: Log cleared. -------------------------------------------------------------... [17:58:11] DBA, Labs, Tool-Labs, Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438354 (Bugreporter) Running it as a normal user will flood recent change, See https://www.wikidata.org/wiki/Wikidata:Administrators%27_noticeboard/Archive/2014/05#Flooding_of_Spe... [18:03:37] DBA, Labs, Tool-Labs, Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438377 (Bugreporter) I try to limit the negative effect of running the tools. At the beginning at most 6-7 tabs are runs. Then I keep only one tab after warning. Now I use two ta... [18:04:42] DBA, Labs, Tool-Labs, Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2437677 (Bugreporter) oops this user is not active at phabricator. [18:08:36] DBA, Labs, Tool-Labs, Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438413 (Magnus) " 6-7 tabs are run" I believe we found the root problem :-) [18:10:23] DBA, Labs, Tool-Labs, Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438422 (Bugreporter) This is previously, as I didn't know how many tabs should be run at most, and what problem would occur when too many tabs are running. [18:10:37] DBA, Labs, Tool-Labs, Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438423 (jcrespo) I think we agreed to use only one "tab" at a time to follow API:Etiquette. I will block all your queries if they continue producing errors in the next 10 minutes,... [18:13:01] DBA, Labs, Tool-Labs, Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438430 (Bugreporter) Currently (and since 20+ minute ago) only one is running. [18:15:20] DBA, Labs, Tool-Labs, Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2437677 (jcrespo) Open>Resolved a:jcrespo Thank you. I see lower amount of errors in the last 20 minutes. I will be monitoring the logs in case the errors return. [18:22:05] DBA, Labs, Tool-Labs, Wikidata: Petscan is being used with excesive parallelism by a user on Wikidata - https://phabricator.wikimedia.org/T139618#2438527 (jcrespo) [18:25:30] DBA, Operations, ops-codfw: pc2006 down - https://phabricator.wikimedia.org/T139283#2438530 (jcrespo) I see no errors either on the web interface. Should we plan a general upgrade of all affected machines, or should we wait in case it fails again? [18:58:54] DBA, Operations, ops-codfw: pc2006 down - https://phabricator.wikimedia.org/T139283#2438642 (Papaul) I think we can plan a general upgrade since it takes not more than 5 minutes to do the upgrade on a system. I will check and see how many systems are affected. [19:09:45] DBA, Community-Tech, MediaWiki-extensions-PageAssessments, Reports-bot, Schema-change: Create tables for PageAssessments in enwiki database - https://phabricator.wikimedia.org/T139552#2438752 (Harej) [19:19:00] DBA, Operations, ops-codfw: pc2006 down - https://phabricator.wikimedia.org/T139283#2438784 (Papaul) please see below for servers that we need to upgrade This affects all PowerEdge R730 and R630 es2011 es2012 es2013 es2014 es2015 es2016 es2017 done es2018 es2019 done pc2004 pc2005 pc2006 done [23:20:36] DBA, Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2440120 (Legoktm)