[05:17:57] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Possibly BBU issues on db1067 - https://phabricator.wikimedia.org/T194852#4221716 (10Marostegui) ``` root@db1067:~# megacli -AdpBbuCmd -a0 | grep Temper Temperature: 47 C Temperature : OK ``` [05:25:23] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Possibly BBU issues on db1067 - https://phabricator.wikimedia.org/T194852#4221721 (10Marostegui) 05Open>03Resolved I have repooled this host. It didn't have any issues after many days and many reboots. So it was probably a one time thing. Resolving... [05:38:09] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-release-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), 10Patch-For-Review: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273#4221725 (10Marostegui) a:03Marostegui I need to deploy other schema changes on... [05:38:35] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop 'tmp1' index from wb_terms table in production - https://phabricator.wikimedia.org/T194270#4221728 (10Marostegui) a:03Marostegui I need to deploy other schema changes on s8, so I will include this as it is a pretty straightforward one. [05:48:35] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-release-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), 10Patch-For-Review: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273#4221731 (10Marostegui) @Ladsgroup can you confirm which indexes we have to drop?... [08:26:05] Amir1: the logging table on wikidatawiki went from 440G to 4G <3 <3 <3 [08:30:17] 10Blocked-on-schema-change, 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10User-Addshore: Change DEFAULT 0 for rev_text_id on production DBs - https://phabricator.wikimedia.org/T190148#4221915 (10Marostegui) [08:30:33] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Schema change for rc_namespace_title_timestamp index - https://phabricator.wikimedia.org/T191519#4221916 (10Marostegui) [08:30:36] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4221917 (10Marostegui) [12:17:32] 10Blocked-on-schema-change, 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10User-Addshore: Change DEFAULT 0 for rev_text_id on production DBs - https://phabricator.wikimedia.org/T190148#4222207 (10Marostegui) s8 eqiad progress: [] dbstore1002 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] db1095 [... [12:18:06] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Schema change for rc_namespace_title_timestamp index - https://phabricator.wikimedia.org/T191519#4222208 (10Marostegui) s8 eqiad progress: [] dbstore1002 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] db... [12:18:30] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4222209 (10Marostegui) s8 eqiad progress: [] dbstore1002 [] labsdb1011 [] labsdb1010 [] labsdb1... [13:39:32] 10DBA, 10Operations: Wikipedia and meta wiki are loading very slowly - https://phabricator.wikimedia.org/T195293#4222312 (10Urbanecm) Confirming from my country (Czech Republic). @addshore told there are some DB errors. [13:39:51] 10DBA, 10Operations: Wikipedia and meta wiki are loading very slowly - https://phabricator.wikimedia.org/T195293#4222298 (10Xaosflux) 503 errors loading various projects: Request from 12.15.146.254 via cp1055 cp1055, Varnish XID 593143055 Error: 503, Backend fetch failed at Tue, 22 May 2018 13:39:08 GMT [13:40:14] 10DBA, 10Operations: Wikipedia and meta wiki are loading very slowly - https://phabricator.wikimedia.org/T195293#4222316 (10Paladox) Im from the United Kingdom. [13:40:31] 10DBA, 10Operations: Wikipedia and meta wiki are loading very slowly - https://phabricator.wikimedia.org/T195293#4222298 (10Marostegui) You've got some examples of the DB errors? [13:40:54] 10DBA, 10Operations: Wikipedia and meta wiki are loading very slowly - https://phabricator.wikimedia.org/T195293#4222319 (10Urbanecm) >>! In T195293#4222314, @Xaosflux wrote: > 503 errors loading various projects: > > Request from <> via cp1055 cp1055, Varnish XID 593143055 > Error: 503, Backend fetc... [13:41:09] 10DBA, 10Operations: 503 error attempting to open multiple projects (Wikipedia and meta wiki are loading very slowly) - https://phabricator.wikimedia.org/T195293#4222320 (10Xaosflux) [13:45:31] 10DBA, 10Operations: 503 error attempting to open multiple projects (Wikipedia and meta wiki are loading very slowly) - https://phabricator.wikimedia.org/T195293#4222298 (10stjn) Same in Russian Wikipedia (from Russia): ``` Request from […] via cp1054 cp1054, Varnish XID 382637319 Error: 503, Backend fetch fai... [13:47:13] 10DBA, 10Operations, 10Wikimedia-log-errors: 503 error attempting to open multiple projects (Wikipedia and meta wiki are loading very slowly) - https://phabricator.wikimedia.org/T195293#4222361 (10zhuyifei1999) [13:47:34] 10DBA, 10Operations, 10Wikimedia-log-errors: 503 error attempting to open multiple projects (Wikipedia and meta wiki are loading very slowly) - https://phabricator.wikimedia.org/T195293#4222298 (10fdans) @Xaosflux be aware that editing the comment doesn't erase the IP info. It's still visible in "View edit h... [13:48:54] 10DBA, 10Operations, 10Wikimedia-log-errors: 503 error attempting to open multiple projects (Wikipedia and meta wiki are loading very slowly) - https://phabricator.wikimedia.org/T195293#4222298 (10Xaosflux) I'm aware - it's no big deal. [13:59:10] 10DBA, 10Operations, 10Wikimedia-log-errors: 503 error attempting to open multiple projects (Wikipedia and meta wiki are loading very slowly) - https://phabricator.wikimedia.org/T195293#4222372 (10Paladox) p:05Unbreak!>03High [14:08:14] 10DBA, 10Operations, 10Wikimedia-Incident, 10Wikimedia-log-errors: 503 error attempting to open multiple projects (Wikipedia and meta wiki are loading very slowly) - https://phabricator.wikimedia.org/T195293#4222383 (10Marostegui) Things have recovered - the DBs errors, so far, look like a consequence and... [15:41:54] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2064 crashed - https://phabricator.wikimedia.org/T195228#4222666 (10Papaul) a:05Papaul>03Marostegui @Marostegui using the power button on the server to power the server doesn't work. Draining the power from the server didn't help as well The serv... [15:43:13] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2064 crashed - https://phabricator.wikimedia.org/T195228#4222671 (10Marostegui) Can we try to swap its PSU with another server from the ones we've decommissioned? Are those compatibles? [15:49:09] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2064 crashed - https://phabricator.wikimedia.org/T195228#4222681 (10Marostegui) From my chat with @Papaul - We have no compatible PSUs from the servers that were decommissioned (they are different vendors) - Changing the power socket/cable didn't ha... [15:49:15] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2064 crashed - https://phabricator.wikimedia.org/T195228#4222682 (10Papaul) @Marostegui no there are not [16:01:42] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install db209[45].codfw.wmnet (sanitarium expansion) - https://phabricator.wikimedia.org/T194781#4222724 (10Papaul) @Marostegui let me know if this racking proposal works for you db2094 row A rack A6 db2095 row C rack C6 [16:03:35] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2064 crashed - https://phabricator.wikimedia.org/T195228#4222726 (10Marostegui) So, looks like this server is lost for good. We have no other similar servers decommissioned we cannot replace spares pieces. Our DCOps suggestion is to basically decommi... [16:03:41] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install db209[45].codfw.wmnet (sanitarium expansion) - https://phabricator.wikimedia.org/T194781#4222728 (10jcrespo) @Papaul, that will work, only requirement is hosts being on separate rows. [16:05:39] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install db209[45].codfw.wmnet (sanitarium expansion) - https://phabricator.wikimedia.org/T194781#4222733 (10Papaul) @jcrespo Thanks [16:13:29] 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: db2064 crashed and totally broken - decommission it - https://phabricator.wikimedia.org/T195228#4222739 (10Marostegui) [16:25:07] 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: db2064 crashed and totally broken - decommission it - https://phabricator.wikimedia.org/T195228#4222766 (10Marostegui) [16:25:35] 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: db2064 crashed and totally broken - decommission it - https://phabricator.wikimedia.org/T195228#4220093 (10Marostegui) [16:35:45] 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: db2064 crashed and totally broken - decommission it - https://phabricator.wikimedia.org/T195228#4222815 (10Marostegui) [16:38:08] 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: db2064 crashed and totally broken - decommission it - https://phabricator.wikimedia.org/T195228#4222836 (10Marostegui) a:05Marostegui>03RobH [16:39:04] 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: db2064 crashed and totally broken - decommission it - https://phabricator.wikimedia.org/T195228#4220093 (10Marostegui) This system is now ready to be decommissioned :-( [17:10:32] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop 'tmp1' index from wb_terms table in production - https://phabricator.wikimedia.org/T194270#4222994 (10Marostegui) Deletion progress: [x] codfw eqiad: [] dbstore1002 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] db1095 [] db1109 [] db1104 []... [21:00:30] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-release-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), 10Patch-For-Review: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273#4223543 (10Ladsgroup) This two index should stay and everything else needs to go...