[00:23:05] 10DBA, 10CheckUser, 10Patch-For-Review: The "show ip" action should also provide a distinct list of user-agents for each IP - https://phabricator.wikimedia.org/T170508 (10Huji) I understand it better now. Let me update the patch and provide you with some queries. [09:27:10] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org, and 2 others: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10Banyek) All those wikis sanitarium set up on db2094 too [09:28:34] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org, and 2 others: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10jcrespo) > All those wikis sanitarium set up on db2094 too Thanks, I can see the triggers there now. Did you run the check script successfully, too? [09:29:42] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org, and 2 others: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10Banyek) yes, of course - I didn't saved the output. :( [12:03:16] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org, and 2 others: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10jcrespo) In theory the drops finished, but I need to do an additonal pass to check for missing hosts/dbs as well as check/remove filters. [12:37:05] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org, and 2 others: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10jcrespo) No filters left that I can see: ``` ./software/dbtools/section s5 | while read host; do echo $host; mysql.py -h $host -e "SHOW ALL SLAVES STATUS\... [12:39:13] 10DBA, 10Operations, 10Epic, 10Patch-For-Review: DB meta task for next DC failover issues - https://phabricator.wikimedia.org/T189107 (10jcrespo) [12:39:18] 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org, and 2 others: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10jcrespo) 05Open>03Resolved Everthing at T184805#4654953 done, except the GTID handling, which has to be checked separately for other reasons. [14:29:17] 10DBA, 10User-Banyek: dbstore2002 tables compression status check - https://phabricator.wikimedia.org/T204930 (10Banyek) 05Open>03Resolved All the sections are compressed, except s3 - but due the number of the tables we have there the compression would take more effort than what we would win with it: I thi... [14:36:17] 10DBA, 10User-Banyek: prometheus-mysqld-exporter not starts on db1117 - https://phabricator.wikimedia.org/T208151 (10jcrespo) 05Open>03Resolved a:03jcrespo [14:38:36] 10DBA, 10Operations, 10ops-eqiad: db1117 went away - https://phabricator.wikimedia.org/T208150 (10jcrespo) There is nothing else left for DBAs here except waiting for errors. [14:38:57] 10DBA, 10Operations, 10ops-eqiad: db1117 went away - https://phabricator.wikimedia.org/T208150 (10jcrespo) p:05High>03Normal [14:43:05] 10DBA, 10Operations: BBU Fail on dbstore2002 - https://phabricator.wikimedia.org/T208320 (10Banyek) [14:57:32] 10DBA, 10Lexicographical data, 10Wikidata, 10Datacenter-Switchover-2018, and 5 others: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared") - https://phabricator.wikimedia.org/T206743 (10jcrespo) In particular, I see https://www.wikidata.org/wiki/Q20... [15:01:05] 10DBA, 10Lexicographical data, 10Wikidata, 10Datacenter-Switchover-2018, and 5 others: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared") - https://phabricator.wikimedia.org/T206743 (10Addshore) 05Open>03Resolved [15:02:16] 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Banyek) [15:02:44] 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Banyek) p:05Triage>03Low [16:04:16] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Banyek: rack/setup/install pc2007-pc2010 - https://phabricator.wikimedia.org/T207259 (10Papaul) [16:11:52] 10DBA, 10Growth-Team, 10Operations, 10StructuredDiscussions, and 2 others: Setup separate logical External Store for Flow in production - https://phabricator.wikimedia.org/T107610 (10kostajh) > I will make sure to check we have fresh backups by then. @jcrespo wanted to check if there are fresh backups. @... [17:48:24] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Banyek: rack/setup/install pc2007-pc2010 - https://phabricator.wikimedia.org/T207259 (10Papaul) [20:42:55] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Banyek: rack/setup/install pc2007-pc2010 - https://phabricator.wikimedia.org/T207259 (10Dzahn) [authdns1001:~] $ host pc2007.codfw.wmnet pc2007.codfw.wmnet has address 10.192.0.104 [authdns1001:~] $ host pc2008.codfw.wmnet pc2008.codfw.wmnet h... [20:45:38] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Banyek: rack/setup/install pc2007-pc2010 - https://phabricator.wikimedia.org/T207259 (10Dzahn) [21:16:36] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Banyek: rack/setup/install pc2007-pc2010 - https://phabricator.wikimedia.org/T207259 (10Papaul) [22:04:24] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-Banyek: rack/setup/install pc2007-pc2010 - https://phabricator.wikimedia.org/T207259 (10Papaul) [23:36:21] 10DBA, 10MediaWiki-Watchlist, 10Growth-Team (Current Sprint), 10MW-1.33-notes (1.33.0-wmf.1; 2018-10-23), and 2 others: Deleting large watchlist takes > 4 seconds causing rollback due to write time limit - https://phabricator.wikimedia.org/T171898 (10Etonkovidova) 05Open>03Resolved The last error was o...