[04:56:45] 10DBA, 06Collaboration-Team-Triage, 10Notifications, 07Schema-change: Temporary index for Echo backfillReadBundles.php? - https://phabricator.wikimedia.org/T137100#2655039 (10Mattflaschen-WMF) 05Open>03declined [06:36:14] 10DBA, 06Operations: Drop PovWatch extension-related database tables from Wikimedia wikis - https://phabricator.wikimedia.org/T54924#2655144 (10Marostegui) There is a backup for these tables at: ``` root@dbstore1001:/srv/tmp/povwatch_tables# ls -lh total 12K -rw-r--r-- 1 root root 860 Sep 13 13:47 s1_povwatc... [07:15:42] 10DBA, 06Operations: Drop PovWatch extension-related database tables from Wikimedia wikis - https://phabricator.wikimedia.org/T54924#2655161 (10Marostegui) The renamed tables have been removed from all the hosts across S1, S3 and S4 [07:15:51] 10DBA, 06Operations: Drop PovWatch extension-related database tables from Wikimedia wikis - https://phabricator.wikimedia.org/T54924#2655162 (10Marostegui) 05Open>03Resolved [07:15:54] 10DBA, 07Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2655163 (10Marostegui) [07:21:49] 10DBA, 06Operations: dbstore2001: crashed - https://phabricator.wikimedia.org/T146259#2655173 (10Marostegui) [07:22:20] 10DBA, 06Operations: dbstore2001: crashed - https://phabricator.wikimedia.org/T146259#2655185 (10Marostegui) [07:23:28] 10DBA: Fix dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T130128#2655190 (10jcrespo) [07:23:30] 10DBA, 06Operations: dbstore2001: crashed - https://phabricator.wikimedia.org/T146259#2655192 (10jcrespo) [07:24:55] I see :-) [07:33:43] 10DBA: Reimage dbstore2002 as jessie - https://phabricator.wikimedia.org/T146261#2655213 (10jcrespo) [07:34:39] 10DBA: Reimage dbstore2002 as jessie - https://phabricator.wikimedia.org/T146261#2655227 (10jcrespo) *hint* [07:35:05] 10DBA: Reimage dbstore2002 as jessie - https://phabricator.wikimedia.org/T146261#2655232 (10Marostegui) a:03Marostegui [07:38:12] 10DBA, 06Labs, 10Labs-Infrastructure, 07Upstream: db1069: convert user_groups table to InnoDB across all the wikis - https://phabricator.wikimedia.org/T146121#2655239 (10Marostegui) ``` root@db1069:/srv# find . -name user_groups.frm | awk -F "." '{print $3}' | awk -F "/" '{print $1}' | uniq -c | sort -k2... [07:52:04] 10DBA: Reimage dbstore2002 as jessie - https://phabricator.wikimedia.org/T146261#2655276 (10jcrespo) Maybe this should be dbstore2001 (both are broken)? Whatever is in a worse state. [07:52:22] 10DBA: Reimage dbstore2001 as jessie - https://phabricator.wikimedia.org/T146261#2655277 (10jcrespo) [07:57:55] 10DBA, 06Operations: Investigate db1082 crash - https://phabricator.wikimedia.org/T145533#2655281 (10MoritzMuehlenhoff) I noticed T141756 which could be related (since db1082 also has the hardware and the oops looked I/O controller-related) [08:03:17] 10DBA, 06Operations: Investigate db1082 crash - https://phabricator.wikimedia.org/T145533#2655283 (10Marostegui) Interesting...feel free to upgrade that firmware if you want. The box isn't pooled yet. [08:04:49] marostegui: I don't think that can be done remotely, the original ticket was tagged ops-eqiad, so this likely need to be upgraded by Chris [08:06:51] moritzm: gotcha. Do you want me to talk to him or you do it? I can coordinate that with him [08:09:25] better avoid the middleman and talk to him directly, he's cmjohnson on IRC, but he doesn't use bouncer, so he's only online starting with the European afternoon [08:10:10] moritzm: Coolio :) [08:10:16] moritzm: Will do [08:10:44] 10DBA, 06Operations: Investigate db1082 crash - https://phabricator.wikimedia.org/T145533#2655291 (10Marostegui) I will coordinate with @Cmjohnson to get this upgraded before we repool it back [08:10:50] Thanks :) [08:11:42] yw, just trying to prove my original point that it's an I/O controller issue :-) [08:13:33] At this point I think it is the only thing we suspect about :) [08:25:44] 10DBA: Investigate (and if possible drop _counters) - https://phabricator.wikimedia.org/T145487#2655321 (10Marostegui) I have renamed it to `TO_DROP__counters` in eqiad hosts ``` dbstore1001.eqiad.wmnet dbstore1002.eqiad.wmnet labsdb1001.eqiad.wmnet labsdb1003.eqiad.wmnet db1069.eqiad.wmnet db1015.eqiad.wmnet d... [09:00:52] 10DBA, 06Operations: db1019: Decommission - https://phabricator.wikimedia.org/T146265#2655380 (10Marostegui) [09:28:06] 10DBA, 06Operations: db1019: Decommission - https://phabricator.wikimedia.org/T146265#2655439 (10Marostegui) This server was doing nothing but replicating, so mysql has been stopped: ``` MariaDB PRODUCTION s4 localhost (none) > show processlist; +-----------+-----------------+------------------+--------------... [10:48:17] 10DBA: Defragment db1070, db1082, db1087, db1092 - https://phabricator.wikimedia.org/T137191#2655586 (10Marostegui) I am testing the alters in db1082 as it is not pooled due to: T145533 [12:32:48] 07Blocked-on-schema-change, 06Community-Tech, 13Patch-For-Review, 07Schema-change, 05WMF-deploy-2016-09-13_(1.28.0-wmf.19): Add local_user_id and global_user_id fields to localuser table in centralauth database - https://phabricator.wikimedia.org/T141951#2655770 (10Marostegui) Only the master is pending... [16:51:02] 10DBA, 10Flow, 03Collab-Team-Q1-July-Sep-2016, 13Patch-For-Review: Cleanup ptwikibooks conversion - https://phabricator.wikimedia.org/T119509#2656488 (10He7d3r) Just to confirm: is it supposed to be in this state right now? https://pt.wikibooks.org/w/index.php?title=Discuss%C3%A3o:Introdu%C3%A7%C3%A3o_%C3%... [18:46:29] 10DBA, 10RESTBase-Cassandra, 06Services-next, 13Patch-For-Review: Import page restrictions to Cassandra restriction table - https://phabricator.wikimedia.org/T135278#2293930 (10Pchelolo) @jcrespo The script at https://gerrit.wikimedia.org/r/#/c/289337/ is ready to be run. We'd need to run it one time over... [22:29:48] 10DBA, 10Flow, 03Collab-Team-Q1-July-Sep-2016, 13Patch-For-Review: Cleanup ptwikibooks conversion - https://phabricator.wikimedia.org/T119509#2657719 (10Mattflaschen-WMF) This is a special case, because the user created the section directly on the page: https://pt.wikibooks.org/w/index.php?title=Discuss%C3... [23:34:41] 10DBA, 10RESTBase-Cassandra, 06Services-next, 13Patch-For-Review: Import page restrictions to Cassandra restriction table - https://phabricator.wikimedia.org/T135278#2657909 (10GWicke) I started a dump using this script on mw1017. This is creating a JSON file per project, with up to 100 restricted revision...