[00:38:21] 10DBA, 10MediaWiki-Database, 10MediaWiki-Documentation, 10Documentation, 10TechCom-RfC (ArchCom-Approved): Bump MediaWiki's minimum supported MySQL Version to 5.5.8 - https://phabricator.wikimedia.org/T161232#3526855 (10Krinkle) [03:45:38] I want to empty my bot's watchlist it shows server error. can anyone help me to empty it? [03:45:48] my bot's name is user:Fawikibot [03:45:54] I tried Special:EditWatchlist/clear but it shows server error [03:46:01] Request from 5.53.33.140 via cp3030 cp3030, Varnish XID 218464776 Error: 503, Service Unavailable at Wed, 16 Aug 2017 03:33:16 GMT [03:46:14] would you please remove it from database? [06:15:56] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3526978 (10Marostegui) When do you want to do this? [06:17:06] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3526980 (10Marostegui) a:03Marostegui [06:19:07] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3235290 (10Marostegui) I am going to start to get ready to run pt-table-checksum on the following tables: revision, pagelinks, templatelinks, text [07:24:36] 10DBA, 10Patch-For-Review: duplicate key problems - https://phabricator.wikimedia.org/T151029#3527052 (10Marostegui) The following tables have been reimported in db2076 and compressed successfully: - frwiki: linter, user_properties, watchlist, wbc_entity_usage - jawiki: user_properties - ruwiki: linter,user_p... [07:44:52] 10DBA, 10Patch-For-Review: Productionize 22 new codfw database servers - https://phabricator.wikimedia.org/T170662#3527071 (10Marostegui) [08:27:00] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3527109 (10Marostegui) [08:27:20] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3363748 (10Marostegui) s6 has been imported [08:43:14] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3527128 (10Marostegui) [08:43:17] 10DBA, 10Readers-Web-Backlog (Tracking): Drop WikiGrok tables from production - https://phabricator.wikimedia.org/T172020#3527126 (10Marostegui) 05Open>03Resolved Dropped from the places it existed [09:41:02] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3527198 (10jcrespo) I don't think it is worth it to run pt-table-checksum on s3 right now: https://logstash.wikimedia.org/goto/e7f445600a027bae3e6633ca00a71172 We should just stop db1015, db1028 and db1031, plus one of the new hosts,... [09:47:35] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3527202 (10Marostegui) >>! In T164488#3527198, @jcrespo wrote: > I don't think it is worth it to run pt-table-checksum on s3 right now: https://logstash.wikimedia.org/goto/e7f445600a027bae3e6633ca00a71172 None of those are s3, why wo... [10:28:55] 10DBA: dbstore1001 mysql crashed with: semaphore wait has lasted > 600 seconds - https://phabricator.wikimedia.org/T169050#3527262 (10Marostegui) 05Open>03Resolved I am going to close this, as this will only be solved once we have migrated this host to multi-instance. [10:37:48] 10DBA, 10Patch-For-Review: Productionize 22 new codfw database servers - https://phabricator.wikimedia.org/T170662#3527289 (10Marostegui) [12:27:35] 10DBA, 10Wikidata, 10Patch-For-Review: Populate term_full_entity_id on www.wikidata.org - https://phabricator.wikimedia.org/T171460#3527409 (10Ladsgroup) I confirm after deploying wmf.14 the maintainance script becomes way faster and also won't remove any labels from testwikidata. i.e. we can run the script... [12:46:48] 10DBA, 10Operations, 10ops-eqiad: RAID crashed on db1078 - https://phabricator.wikimedia.org/T173365#3527467 (10Cmjohnson) A case for a new disk has been created. Your case was successfully submitted. Please note your Case ID: 5322179480 for future reference. Regarding the DB crash because of 1 disk failur... [12:47:44] 10DBA: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3527469 (10Zoranzoki21) Not need [12:48:15] 10DBA, 10Operations, 10ops-eqiad: RAID crashed on db1078 - https://phabricator.wikimedia.org/T173365#3527471 (10Marostegui) >>! In T173365#3527467, @Cmjohnson wrote: > A case for a new disk has been created. Your case was successfully submitted. Please note your Case ID: 5322179480 for future reference. >... [12:49:46] 10DBA, 10Operations, 10ops-eqiad: RAID crashed on db1078 - https://phabricator.wikimedia.org/T173365#3527487 (10Cmjohnson) Sure, sometime in the next few hours is fine. [12:59:46] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3527516 (10Marostegui) I have run the first diff for all the revision tables between db1078 and db1015 and there are no differences. Next table to be checked: pagelinks [13:05:17] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3527518 (10Marostegui) [13:20:20] 10DBA, 10Operations, 10ops-eqiad: RAID crashed on db1078 - https://phabricator.wikimedia.org/T173365#3527550 (10jcrespo) > Regarding the DB crash because of 1 disk failure is odd. I will need to take the server offline. Let me know when it's safe to do so. Joe's thesis is that maybe smartpath setup doesn't... [13:50:59] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3527611 (10MarcoAurelio) If you're going to be avalaible in the next hours we can try to do it if you think it's okay and have time. [13:52:40] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3527615 (10Marostegui) >>! In T173159#3527611, @MarcoAurelio wrote: > If you're going to be avalaible in the next hours we can try to do it if you think it's okay and have time. In which TZ are you? I... [13:59:58] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3527619 (10RuyP) If it helps, my timezone is -3 and I am around almost all day long lately... [14:01:50] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3527621 (10Marostegui) >>! In T173159#3527619, @RuyP wrote: > If it helps, my timezone is -3 and I am around almost all day long lately... Thanks - do you want to try 11AM UTC tomorrow? [14:04:08] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3527624 (10MarcoAurelio) I give way to @RuyP to handle this with your coordination. [14:04:48] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3527629 (10RuyP) >>! In T173159#3527621, @Marostegui wrote: >>>! In T173159#3527619, @RuyP wrote: >> If it helps, my timezone is -3 and I am around almost all day long lately... > > Thanks - do you wa... [14:07:50] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3527638 (10Marostegui) >>! In T173159#3527629, @RuyP wrote: >>>! In T173159#3527621, @Marostegui wrote: >>>>! In T173159#3527619, @RuyP wrote: >>> If it helps, my timezone is -3 and I am around almost... [14:28:00] 10DBA, 10Operations, 10Wikimedia-Site-requests, 10User-MarcoAurelio: 13 global renames stuck - https://phabricator.wikimedia.org/T173419#3527677 (10MarcoAurelio) [14:34:50] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3527702 (10RuyP) >>! In T173159#3527638, @Marostegui wrote: >>>! In T173159#3527629, @RuyP wrote: >>>>! In T173159#3527621, @Marostegui wrote: >>>>>! In T173159#3527619, @RuyP wrote: >>>> If it helps,... [14:35:25] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3527703 (10Marostegui) >>! In T173159#3527702, @RuyP wrote: >>>! In T173159#3527638, @Marostegui wrote: >>>>! In T173159#3527629, @RuyP wrote: >>>>>! In T173159#3527621, @Marostegui wrote: >>>>>>! In T... [14:42:03] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3527711 (10RuyP) >>! In T173159#3527703, @Marostegui wrote: >>>! In T173159#3527702, @RuyP wrote: >>>>! In T173159#3527638, @Marostegui wrote: >>>>>! In T173159#3527629, @RuyP wrote: >>>>>>! In T173159... [14:43:38] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3527712 (10Marostegui) Thank you! [14:51:37] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3527734 (10MarcoAurelio) Please do not action this rename until {T173419} is fixed and its reasons known. Thank you. [14:53:33] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3527741 (10RuyP) >>! In T173159#3527734, @MarcoAurelio wrote: > Please do not action this rename until {T173419} is fixed and its reasons known. Thank you. Thanks for letting me know, Marco. :) [14:55:25] 10DBA, 10Wikimedia-Site-requests: Rename user Darwinius to DarwIn - https://phabricator.wikimedia.org/T173159#3527743 (10Marostegui) >>! In T173159#3527741, @RuyP wrote: >>>! In T173159#3527734, @MarcoAurelio wrote: >> Please do not action this rename until {T173419} is fixed and its reasons known. Thank you.... [15:12:23] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Wikidata, and 5 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3527782 (10Marostegui) @hoo this happened again just now. As we talked during wikimania here is the... [15:13:29] buf [15:27:49] 10DBA, 10Wikimedia-Site-requests, 10Tracking: Database table cleanup (tracking) - https://phabricator.wikimedia.org/T18660#3527815 (10Marostegui) [15:27:51] 10DBA, 10ContentTranslation, 10Language-2017 Sprint 10, 10WorkType-Maintenance: Remove cx_drafts table from production - https://phabricator.wikimedia.org/T172364#3527812 (10Marostegui) 05Open>03Resolved I have checked and this table does not exist on any of the production masters. [15:32:46] 10DBA, 10ContentTranslation, 10Language-2017 Sprint 10, 10WorkType-Maintenance: Remove cx_drafts table from production - https://phabricator.wikimedia.org/T172364#3527854 (10jcrespo) 05Resolved>03Open Sorry? ``` root@db1031[wikishared]> SHOW TABLES; +----------------------+ | Tables_in_wikishared | +--... [15:32:49] 10DBA, 10Wikimedia-Site-requests, 10Tracking: Database table cleanup (tracking) - https://phabricator.wikimedia.org/T18660#3527857 (10jcrespo) [15:34:01] 10DBA, 10ContentTranslation, 10Language-2017 Sprint 10, 10WorkType-Maintenance: Remove cx_drafts table from production - https://phabricator.wikimedia.org/T172364#3527871 (10Marostegui) >>! In T172364#3527854, @jcrespo wrote: > Sorry? > ``` > root@db1031[wikishared]> SHOW TABLES; > +----------------------+... [15:37:20] 10DBA, 10ContentTranslation, 10Language-2017 Sprint 10, 10WorkType-Maintenance: Remove cx_drafts table from production - https://phabricator.wikimedia.org/T172364#3527904 (10Marostegui) I have renamed the table on db1029 and will leave it like that for a few days to make sure nothing complains/breaks ``` r... [15:46:50] 10DBA, 10ContentTranslation, 10Language-2017 Sprint 10, 10WorkType-Maintenance: Remove cx_drafts table from production - https://phabricator.wikimedia.org/T172364#3527941 (10KartikMistry) I did some random checks and draft loading in ContentTranslation seems OK. [16:14:47] 10DBA, 10Operations, 10media-storage, 10monitoring: icinga hp raid check timeout on busy ms-be and db machines - https://phabricator.wikimedia.org/T141252#3528035 (10herron) [16:19:10] 10DBA, 10Operations, 10media-storage, 10monitoring: icinga hp raid check timeout on busy ms-be and db machines - https://phabricator.wikimedia.org/T141252#3528044 (10jcrespo) Should we close this too, or too early to say? @herron [16:22:57] 10DBA, 10ContentTranslation, 10Language-2017 Sprint 10, 10WorkType-Maintenance: Remove cx_drafts table from production - https://phabricator.wikimedia.org/T172364#3528058 (10Marostegui) >>! In T172364#3527941, @KartikMistry wrote: > I did some random checks and draft loading in ContentTranslation seems OK.... [16:23:27] 10DBA, 10Operations, 10media-storage, 10monitoring: icinga hp raid check timeout on busy ms-be and db machines - https://phabricator.wikimedia.org/T141252#3528060 (10herron) 05Open>03Resolved a:03herron Sure, sounds good to me. We could always reopen and evaluate if the issue occurs again in the fut... [16:33:22] 10DBA, 10ContentTranslation, 10Language-2017 Sprint 10, 10WorkType-Maintenance: Remove cx_drafts table from production - https://phabricator.wikimedia.org/T172364#3528086 (10Marostegui) By the way, I have backed up this table at: ``` dbstore1001:/srv/tmp/T172364/cx_drafts.sql ``` [16:41:50] 10DBA, 10Operations, 10ops-eqiad: RAID crashed on db1078 - https://phabricator.wikimedia.org/T173365#3528105 (10Cmjohnson) I verified the settings, everything appears normal. [17:26:15] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Wikidata, and 5 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3224974 (10thcipriani) >>! In T164173#3527782, @Marostegui wrote: > @hoo this happened again just n... [17:45:45] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Wikidata, and 5 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3528397 (10greg) [17:46:10] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Wikidata, and 5 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3224976 (10greg) [18:04:20] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3528472 (10Bawolff) [18:05:05] 10DBA, 10Wikimedia-Site-requests, 10Tracking: Database table cleanup (tracking) - https://phabricator.wikimedia.org/T18660#3528478 (10Bawolff) [18:21:57] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Wikidata, and 5 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3528601 (10Marostegui) >>! In T164173#3528312, @thcipriani wrote: >>>! In T164173#3527782, @Maroste... [18:23:32] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3528602 (10Marostegui) The diff between db1015 and db1078 for the pagelinks table revealed no differences. [18:23:41] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3528603 (10Marostegui) [23:23:55] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for bawikibooks - https://phabricator.wikimedia.org/T173473#3529530 (10Urbanecm) [23:26:35] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for bawikibooks - https://phabricator.wikimedia.org/T173473#3529558 (10Urbanecm) 05Open>03Invalid Seems it is about reopening, not creation. No DBA attention should be needed then.