[02:24:24] 10DBA, 10Labs, 10Tool-Labs: Tool Labs logging vs indexed version returning dramatically different results - https://phabricator.wikimedia.org/T168349#3361851 (10zhuyifei1999) [02:30:21] 10DBA, 10Labs, 10Tool-Labs: Tool Labs logging vs indexed version returning dramatically different results - https://phabricator.wikimedia.org/T168349#3361809 (10zhuyifei1999) FWIW, using https://tools.wmflabs.org/tools-info/optimizer.py, the EXPLAIN-s for both queries query is basically the same as (differ a... [02:38:00] 10DBA, 10Labs, 10Tool-Labs: Tool Labs logging vs indexed version returning dramatically different results - https://phabricator.wikimedia.org/T168349#3361856 (10zhuyifei1999) Regarding logging_logindex: | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | | 1 | SIMPLE | l... [03:23:28] 10DBA, 10Labs, 10Tool-Labs: Tool Labs logging vs indexed version returning dramatically different results - https://phabricator.wikimedia.org/T168349#3361908 (10bd808) [03:23:30] 10DBA, 10Labs, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3361909 (10bd808) [03:24:50] 10DBA, 10Labs: enwiki_p logging vs logging_userindex returning dramatically different results - https://phabricator.wikimedia.org/T168349#3361809 (10bd808) [03:26:54] 10DBA, 10Labs, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3361916 (10bd808) Linked {T168349} as a child. The report there is pretty long for pasting into this task. [03:30:14] 10DBA, 10Labs: enwiki_p logging vs logging_userindex returning dramatically different results - https://phabricator.wikimedia.org/T168349#3361928 (10zhuyifei1999) Maybe DBAs have better ideas, but this is an optimised-to-4-min query: {P5596} The relevant EXPLAIN is: | id | select_type | table | type | possib... [03:39:50] 10DBA, 10Operations, 10Goal: Migrate MySQLs to use ROW-based replication - https://phabricator.wikimedia.org/T109179#3361933 (10MZMcBride) What's the status of this task? The previous comment is from over a year ago. [04:02:15] 10DBA, 10Labs: enwiki_p logging vs logging_userindex returning dramatically different results - https://phabricator.wikimedia.org/T168349#3361943 (10MusikAnimal) >>! In T168349#3361928, @zhuyifei1999 wrote: > Maybe DBAs have better ideas, but this is an optimised-to-4-min query: > > P5596 This is amazing. Th... [05:08:21] 10DBA, 10Labs: enwiki_p logging vs logging_userindex returning dramatically different results - https://phabricator.wikimedia.org/T168349#3361969 (10zhuyifei1999) >>! In T168349#3361943, @MusikAnimal wrote: > I think something really funky is going on. The grouping mechanism don't seem to work correctly from... [05:29:41] 10DBA, 10Operations, 10Goal: Migrate MySQLs to use ROW-based replication - https://phabricator.wikimedia.org/T109179#1542524 (10Marostegui) This is an Epic task and quite hard to achieve in short or even medium term. To give you an example, row based replication is quite strict with data drifts and can break... [05:41:10] 10DBA: dbstore2001 s5 thread is 6 days delayed - https://phabricator.wikimedia.org/T168354#3361973 (10Marostegui) [05:41:15] 10DBA: dbstore2001 s5 thread is 6 days delayed - https://phabricator.wikimedia.org/T168354#3361985 (10Marostegui) p:05Triage>03High [05:47:03] 10DBA: dbstore2001 s5 thread is 6 days delayed - https://phabricator.wikimedia.org/T168354#3361989 (10Marostegui) Something started the 12th between 12:20 and 13:00 https://grafana.wikimedia.org/dashboard/db/mysql?panelId=6&fullscreen&orgId=1&var-dc=codfw%20prometheus%2Fops&var-server=dbstore2001&from=1496727827... [06:35:32] 10DBA, 10Operations, 10Goal: Migrate MySQLs to use ROW-based replication - https://phabricator.wikimedia.org/T109179#3362034 (10jcrespo) @MZMcBride Note this doesn't depend on us DBAs- changing to row based replication is a one-time change that is instantaneous. However, the application has to work well with... [06:48:29] 10DBA, 10Operations: Prepare mysql hosts for stretch - https://phabricator.wikimedia.org/T168356#3362038 (10jcrespo) [06:49:23] 10DBA, 10Community-Wikimetrics, 10Icinga, 10Labs, and 2 others: Evaluate future of wmf puppet module "mysql" - https://phabricator.wikimedia.org/T165625#3362051 (10jcrespo) [06:49:27] 10DBA, 10Operations, 10Patch-For-Review: Adapt wmf-mariadb101 package for stretch and adapt its service to systemd - https://phabricator.wikimedia.org/T116903#3362052 (10jcrespo) [06:49:30] 10DBA, 10Operations, 10Patch-For-Review: mysql user and group should be a system user/group - https://phabricator.wikimedia.org/T100501#3362053 (10jcrespo) [06:49:33] 10DBA, 10Operations: Prepare mysql hosts for stretch - https://phabricator.wikimedia.org/T168356#3362050 (10jcrespo) [06:50:12] 10DBA, 10ArchCom-RfC, 10MediaWiki-Database, 10RfC: Should we bump Mediawiki's minimum supported MySQL Version to 5.5? - https://phabricator.wikimedia.org/T161232#3362054 (10jcrespo) [06:50:55] 10DBA, 10ArchCom-RfC, 10MediaWiki-Database, 10RfC: Should we bump Mediawiki's minimum supported MySQL Version to 5.5? - https://phabricator.wikimedia.org/T161232#3125897 (10jcrespo) Renaming because phabricator contains other projects than mediawiki, so contextualizing better- mysql appears on many searche... [06:53:06] 10DBA, 10ArchCom-RfC, 10MediaWiki-Database, 10RfC: Should we bump MediaWiki's minimum supported MySQL Version to 5.5? - https://phabricator.wikimedia.org/T161232#3362074 (10Kghbln) [07:08:55] 10DBA: dbstore2001 s5 thread is 6 days delayed - https://phabricator.wikimedia.org/T168354#3362103 (10Marostegui) Nothing on HW logs either [07:09:56] 10Blocked-on-schema-change, 10DBA: Convert unique keys into primary keys for some wiki tables on s5 - https://phabricator.wikimedia.org/T166207#3362109 (10Marostegui) [07:41:52] 10DBA, 10Patch-For-Review: Run pt-table-checksum on s1 (enwiki) - https://phabricator.wikimedia.org/T162807#3362190 (10Marostegui) I have killed the alter table running on enwiki.revision on db1047. It has been running for 13 days already and only did 58% of progress on it Unfortunately I will need to exclude... [07:50:30] 10DBA, 10Schema-change: Drop titlekey table from all wmf databases - https://phabricator.wikimedia.org/T164949#3362206 (10Marostegui) [08:19:22] 10DBA, 10Schema-change: Drop titlekey table from all wmf databases - https://phabricator.wikimedia.org/T164949#3362251 (10Marostegui) [08:45:42] 10DBA, 10Schema-change: Drop titlekey table from all wmf databases - https://phabricator.wikimedia.org/T164949#3362324 (10Marostegui) [08:51:03] 10DBA, 10Schema-change: Drop titlekey table from all wmf databases - https://phabricator.wikimedia.org/T164949#3362341 (10Marostegui) [08:51:11] 10DBA, 10Schema-change: Drop titlekey table from all wmf databases - https://phabricator.wikimedia.org/T164949#3252028 (10Marostegui) 05Open>03Resolved [08:51:15] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3362343 (10Marostegui) [09:03:24] 10DBA, 10Operations: Drop wikilove_image_log table from Wikimedia wikis - https://phabricator.wikimedia.org/T127219#2036092 (10Marostegui) Current status - it looks like it has been partially deleted (or was never placed) on some wikis: s1: ``` db1052.eqiad.wmnet -rw-rw---- 1 mysql mysql 11M Jan 14 2015 /srv... [09:35:04] 10DBA, 10Operations: Drop wikilove_image_log table from Wikimedia wikis - https://phabricator.wikimedia.org/T127219#3362495 (10Marostegui) I have taken a backup of this tables at: ``` dbstore1001:/srv/tmp/T127219 ``` It is tiny really: ``` root@dbstore1001:/srv/tmp/T127219# pwd /srv/tmp/T127219 root@dbstore... [09:49:08] 10DBA, 10Labs: enwiki_p logging vs logging_userindex returning dramatically different results - https://phabricator.wikimedia.org/T168349#3361809 (10Marostegui) In which hosts did you do the tests? [10:05:08] 10DBA, 10Labs, 10Tool-Labs, 10Tracking: Certain tools users create multiple long running queries that take all memory from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601#3362619 (10jcrespo) [10:06:02] 10DBA, 10Labs, 10Tool-Labs, 10Tracking: Certain tools users create multiple long running queries that take all memory from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601#1830725 (10jcrespo) [10:06:04] 10DBA, 10Labs, 10Tool-Labs: s51053 (tools.jackbot) is abusing resources on labsdbs, throttle his grants - https://phabricator.wikimedia.org/T114559#3362620 (10jcrespo) [10:06:15] 10DBA, 10Labs, 10Tool-Labs: s51053 (tools.jackbot) is abusing resources on labsdbs, throttle his grants - https://phabricator.wikimedia.org/T114559#1699378 (10jcrespo) [10:06:17] 10DBA, 10Labs, 10Tool-Labs, 10Tracking: Certain tools users create multiple long running queries that take all memory from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601#1830725 (10jcrespo) [10:08:00] 10DBA, 10Labs, 10Tool-Labs: s51053 (tools.jackbot) is abusing resources on labsdbs, throttle his grants - https://phabricator.wikimedia.org/T114559#3362637 (10JackPotte) This should be resolved now, do you know a monitoring on which I could check it please? [10:09:29] 10DBA, 10Labs, 10Tool-Labs: s51053 (tools.jackbot) is abusing resources on labsdbs, throttle his grants - https://phabricator.wikimedia.org/T114559#3362642 (10jcrespo) This is resolved, I only edited it because of admin purposes (correct tracking). Sorry for the spam, email is automatic. [10:21:59] what is happening on dbstore2001? [10:22:36] Check this: https://phabricator.wikimedia.org/T168354 [10:23:00] you stopped it on Tue, Jun 20, 07:22 [10:23:05] Yes [10:23:06] but it is unresponsive noe [10:23:13] even if the process is running [10:23:15] check this now: [10:23:28] https://phabricator.wikimedia.org/T165033#3257985 [10:24:08] server is not up [10:24:17] the process is, but it doesn't accept conenctions [10:24:24] yes, it takes around 3:30h for it to be fully up [10:24:27] which is madness [10:24:29] arg [10:24:39] that is not normal [10:24:42] nope [10:24:43] not at all [10:24:57] but as it happened last time, I am "not worried" that I cannot still access it [10:25:05] yes, thank you, [10:25:12] that is the part I was missing [10:25:24] I didn't know if you were doing maintenance or something else [10:25:31] Sorry - i think that happened when you were with some days off (the investigation that led to see that it takes 3.30h) [10:26:21] did you disable events before stopping it or starting it? [10:26:40] Before stopping yes [10:27:08] Not before starting though [10:27:38] It didn't make much difference onthe previous tests (to start with events disabled) [10:27:50] I started with slaves stopped [10:28:27] we shoud just eventually rebuild it with multiple instances [10:28:38] +10000 [11:38:44] 4 hours and still not up [11:38:48] but it is doing stuff [11:38:52] as per strace [13:23:23] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s5 - https://phabricator.wikimedia.org/T166207#3363263 (10Marostegui) db1071 done: ``` root@neodymium:/home/marostegui# for i in `cat s5_tables`; do echo $i; mysql --skip-ssl -hdb1071 wikidata... [13:23:34] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s5 - https://phabricator.wikimedia.org/T166207#3363264 (10Marostegui) [13:23:58] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s5 - https://phabricator.wikimedia.org/T166207#3288328 (10Marostegui) [13:24:37] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s5 - https://phabricator.wikimedia.org/T166207#3288328 (10Marostegui) For those tables existing on db1095, they are done: ``` root@neodymium:/home/marostegui# for i in `cat s5_tables`; do echo... [15:21:21] 10DBA: dbstore2001 s5 thread is 6 days delayed - https://phabricator.wikimedia.org/T168354#3363734 (10Marostegui) There is clearly something wrong with this host. T165033#3257985 we could see that it normally took around 3 hours to get MySQL up after the socket was created, so far it has been almost 8 hours and... [15:22:33] 10DBA, 10Labs, 10User-bd808, 10cloud-services-team (Kanban): setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 - https://phabricator.wikimedia.org/T168021#3353312 (10JAllemandou) Hi @Marostegui I can't connect to `dewiki_p` nor `wikidatawiki_p` on `labsdb-analytics`. Should this task be reopened? [15:23:20] 10DBA, 10Labs, 10User-bd808, 10cloud-services-team (Kanban): setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 - https://phabricator.wikimedia.org/T168021#3363746 (10Marostegui) What errors are you getting? [15:23:48] 10DBA: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3363748 (10Marostegui) [15:24:02] 10DBA: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3363763 (10Marostegui) p:05Triage>03Normal [15:39:40] 10DBA, 10Labs, 10User-bd808, 10cloud-services-team (Kanban): setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 - https://phabricator.wikimedia.org/T168021#3363809 (10Marostegui) I have recreated the views, can you try again? if you can show the error you are getting, that would be helpful. Als... [15:58:03] 10DBA, 10Analytics: Purge all old data from master - https://phabricator.wikimedia.org/T168414#3363866 (10Ottomata) [16:14:32] 10DBA, 103d, 10Multimedia, 10Patch-For-Review: Have search recognise STL files as a new kind of media file ('type:3d' or whatever) - https://phabricator.wikimedia.org/T157348#3363964 (10dr0ptp4kt) @jcrespo we're interested in merging - would you be able to test so as to confirm your code review? [16:19:32] 10DBA: dbstore2001 s5 thread is 6 days delayed - https://phabricator.wikimedia.org/T168354#3363993 (10Marostegui) And it is up: ``` 170620 16:04:28 [Note] /opt/wmf-mariadb10/bin/mysqld: ready for connections ``` So it took 8:30h from the socket creation until the server was fully available. I have started the s... [16:20:53] 10DBA: dbstore2001 takes 3 hours to start MySQL (was: dbstore2001 takes 3 hours to start MySQL after a crash) - https://phabricator.wikimedia.org/T165033#3363996 (10Marostegui) It took now 8:30h to get a normal start: T168354#3363993 [16:25:49] 10DBA, 10Labs, 10User-bd808, 10cloud-services-team (Kanban): setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 - https://phabricator.wikimedia.org/T168021#3364008 (10JAllemandou) I use a script checking available views from `information_schema`. For the moment it still tells me `dewiki` and `... [16:26:25] 10DBA: dbstore2001 s5 thread is 6 days delayed - https://phabricator.wikimedia.org/T168354#3364010 (10Marostegui) I have stopped all the slaves but the s5 one and I will leave it running alone until tomorrow morning, to see if it is able to reduce its lag when it has the whole server just for itself. [16:34:19] 10DBA, 10Labs, 10User-bd808, 10cloud-services-team (Kanban): setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 - https://phabricator.wikimedia.org/T168021#3364022 (10Marostegui) I don't know what that script does but: ``` mysql:root@localhost [information_schema]> select @@hostname; +---------... [17:29:20] 10DBA, 10Labs, 10User-bd808, 10cloud-services-team (Kanban): setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011 - https://phabricator.wikimedia.org/T168021#3364388 (10JAllemandou) I can access the views - Sorry for the false positive. However my script still don't find the DB - I'll need to loo... [18:25:16] 10DBA, 10Labs: enwiki_p logging vs logging_userindex returning dramatically different results - https://phabricator.wikimedia.org/T168349#3364718 (10MusikAnimal) >>! In T168349#3362544, @Marostegui wrote: > In which hosts did you do the tests? Sorry I didn't record this information. I ran `sql enwiki` on `too... [18:35:23] 10DBA, 10Operations: Drop wikilove_image_log table from Wikimedia wikis - https://phabricator.wikimedia.org/T127219#3364802 (10kaldari) @Marostegui: Thanks for the info about the back-up. Might be useful data for some wiki archeologist one day :) [19:03:02] 10DBA, 10Labs, 10Stewards-and-global-tools, 10Tool-Labs: Throttling linkwatcher tool user as it is consuming 100% CPU - https://phabricator.wikimedia.org/T121094#1868898 (10Luke081515) Any updates on this old ticket? [20:30:05] 10DBA, 10Operations, 10Wikimedia-Site-requests, 10Patch-For-Review: Create CoC committee private wiki - https://phabricator.wikimedia.org/T165977#3365315 (10Dereckson) [22:11:15] 10DBA, 10Labs, 10cloud-services-team, 10wikitech.wikimedia.org: move wikitech and labstestwiki to s3 (needs discussion) - https://phabricator.wikimedia.org/T167973#3365576 (10bd808) [22:41:35] 10DBA, 10Operations, 10Goal: Migrate MySQLs to use ROW-based replication - https://phabricator.wikimedia.org/T109179#1542524 (10bd808) >>! In T109179#3365630, @MZMcBride wrote: > In the context of Wikimedia Labs, the word testing is confusing to me. Isn't all of Labs for testing? It's nice to hear that the d...