[06:56:12] 10DBA, 10Data-Services, 10Patch-For-Review, 10User-bd808, and 2 others: Drop ukwikimedia from labsdb hosts (was: ukwikimedia still present on replicas dbs on labs hosts) - https://phabricator.wikimedia.org/T169488#3419403 (10Marostegui) 05Resolved>03Open Hi, The script that checks for data not suppos... [08:06:20] 10DBA: Drop localisation and localisation_file_hash tables, l10nwiki databases too - https://phabricator.wikimedia.org/T119811#3419539 (10Marostegui) Dropped on s2. [08:09:57] 10DBA: include primary key field as secondary index suffix - https://phabricator.wikimedia.org/T105440#3419576 (10Aklapper) a:05Springle>03None [ Resetting assignee as assignee account is not active anymore ] [08:10:04] 10DBA, 10Operations, 10Traffic, 10WMF-Legal, and 2 others: dbtree loads third party resources (from jquery.com and google.com) - https://phabricator.wikimedia.org/T96499#3419578 (10Aklapper) a:05Springle>03None [ Resetting assignee as assignee account is not active anymore ] [08:10:07] 10DBA, 10Cloud-Services, 10Cloud-VPS: Queries of commonswiki_p.filearchive for fa_sha1 are slow - https://phabricator.wikimedia.org/T71088#3419584 (10Aklapper) a:05Springle>03None [ Resetting assignee as assignee account is not active anymore ] [08:12:24] 10DBA: include primary key field as secondary index suffix - https://phabricator.wikimedia.org/T105440#1443918 (10Marostegui) Not sure we are going to work on this any time soon. We have unified the revision table across the board by the way. [08:59:44] 10DBA, 10Analytics: Create a user for the eventlogging_cleaner script on the analytics slaves - https://phabricator.wikimedia.org/T170118#3419829 (10elukey) [09:07:31] 10DBA, 10Data-Services, 10Patch-For-Review, 10User-bd808, and 2 others: Drop ukwikimedia from labsdb hosts (was: ukwikimedia still present on replicas dbs on labs hosts) - https://phabricator.wikimedia.org/T169488#3419867 (10jcrespo) cc @bd808 [10:20:36] hello everybody [10:21:24] I am trying to come up with a path for the eventlogging_cleaner.py user on dbstore1002/db1047 [10:21:44] We are in our weekly meeting [10:22:02] okok I'll keep writing, answer whenver you have time [10:22:04] :) [10:22:07] coolio! [10:22:11] thanks [10:22:43] so I am wondering if I'd need to just have "IDENTIFIED VIA unix_socket" + related my.cnf or something different [10:23:30] like password in the puppet private repo + IDENTIFIED BY + my.cnf etc.. [10:23:41] (T170118) [10:23:41] T170118: Create a user for the eventlogging_cleaner script on the analytics slaves - https://phabricator.wikimedia.org/T170118 [10:24:10] 10DBA, 10Analytics, 10User-Elukey: Create a user for the eventlogging_cleaner script on the analytics slaves - https://phabricator.wikimedia.org/T170118#3420110 (10elukey) [12:19:47] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Performance-Team, and 6 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3420358 (10daniel) @aaron I'm trying to wrap my head around the code in Title::invalidateCa... [12:24:22] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Performance-Team, and 6 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3420378 (10daniel) @jcrespo Looking at T169884, it seems that it's an unrelated issue trigg... [13:03:03] 10DBA, 10Analytics-Kanban, 10User-Elukey: Create a user for the eventlogging_cleaner script on the analytics slaves - https://phabricator.wikimedia.org/T170118#3420477 (10elukey) [13:07:45] 10DBA, 10Operations, 10monitoring, 10Prometheus-metrics-monitoring: Create a script to regenerate prometheus mysqld exporter listing that works with puppetdb - https://phabricator.wikimedia.org/T145072#3420498 (10faidon) [13:08:08] 10DBA, 10Operations, 10monitoring, 10Patch-For-Review, 10Prometheus-metrics-monitoring: MySQL monitoring with prometheus - https://phabricator.wikimedia.org/T143896#3420499 (10faidon) [13:08:21] 10DBA, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 5 others: Drop tables with bad data: mediawiki_page_create_1 mediawiki_revision_create_1 - https://phabricator.wikimedia.org/T169781#3420500 (10elukey) >>! In T169781#3412167, @Marostegui wrote: > @elukey I assume you meant d... [13:31:40] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Performance-Team, and 6 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3420592 (10Ladsgroup) a:05Ladsgroup>03daniel [13:33:14] 10DBA, 10Operations, 10monitoring, 10Patch-For-Review, 10Prometheus-metrics-monitoring: MySQL monitoring with prometheus - https://phabricator.wikimedia.org/T143896#3420611 (10jcrespo) [13:55:06] 10DBA, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 5 others: Drop tables with bad data: mediawiki_page_create_1 mediawiki_revision_create_1 - https://phabricator.wikimedia.org/T169781#3420713 (10Ottomata) Hey ya fine with me! The data is brand new, no one is looking for it e... [13:59:40] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Performance-Team, and 6 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3420723 (10daniel) @aaron another question: does RefreshLinksJob also purge the CDN cache a... [14:19:08] 10DBA, 10Patch-For-Review: Refactor puppet mariadb class to support multi-instance hosts - https://phabricator.wikimedia.org/T169514#3420779 (10jcrespo) The easy part is done, we will now see how to multiplex the monitoring: ``` root@db1096:/etc/mysql/mysqld.conf.d$ cat * [mysqld@s1] datadir=/srv/sqldata.s1 tm... [14:27:14] 10DBA, 10Data-Services, 10Patch-For-Review: Expose ar_content_format and ar_content_model columns of archive table on Labs replicas - https://phabricator.wikimedia.org/T89741#3420806 (10chasemp) >>! In T89741#3416082, @gerritbot wrote: > Change 363851 had a related patch set uploaded (by Umherirrender; owner... [14:54:04] 10DBA, 10Mail, 10Operations: Setup database for dmarc service - https://phabricator.wikimedia.org/T170158#3420991 (10jcrespo) [15:04:06] 10DBA, 10Data-Services, 10Patch-For-Review, 10User-bd808, and 2 others: Drop ukwikimedia from labsdb hosts (was: ukwikimedia still present on replicas dbs on labs hosts) - https://phabricator.wikimedia.org/T169488#3421024 (10bd808) >>! In T169488#3419403, @Marostegui wrote: > Hi, > > The script that chec... [15:06:05] 10DBA, 10Data-Services, 10Patch-For-Review, 10User-bd808, and 2 others: Drop ukwikimedia from labsdb hosts (was: ukwikimedia still present on replicas dbs on labs hosts) - https://phabricator.wikimedia.org/T169488#3421040 (10Marostegui) >>! In T169488#3421024, @bd808 wrote: >>>! In T169488#3419403, @Marost... [15:12:15] 10DBA, 10Data-Services, 10Patch-For-Review, 10User-bd808, and 2 others: Drop ukwikimedia from labsdb hosts (was: ukwikimedia still present on replicas dbs on labs hosts) - https://phabricator.wikimedia.org/T169488#3421074 (10jcrespo) Yes, I only dropped the databases- I thought the script run took care of... [15:23:04] 10DBA, 10Data-Services, 10Patch-For-Review, 10User-bd808, and 2 others: Drop ukwikimedia from labsdb hosts (was: ukwikimedia still present on replicas dbs on labs hosts) - https://phabricator.wikimedia.org/T169488#3421136 (10Marostegui) 05Open>03Resolved I have dropped the views from all the host. The... [15:24:20] 10DBA, 10Mail, 10Operations: Setup database for dmarc service - https://phabricator.wikimedia.org/T170158#3421146 (10jcrespo) [15:24:28] 10DBA, 10Mail, 10Operations: Setup database for dmarc service - https://phabricator.wikimedia.org/T170158#3420991 (10jcrespo) [15:26:47] 10DBA, 10Data-Services, 10Patch-For-Review, 10User-bd808, and 2 others: Drop ukwikimedia from labsdb hosts (was: ukwikimedia still present on replicas dbs on labs hosts) - https://phabricator.wikimedia.org/T169488#3421166 (10jcrespo) ``` root@labsdb1009[(none)]> pager grep ukwikimedia PAGER set to 'grep uk... [15:28:44] 10DBA, 10Data-Services, 10Patch-For-Review, 10User-bd808, and 2 others: Drop ukwikimedia from labsdb hosts (was: ukwikimedia still present on replicas dbs on labs hosts) - https://phabricator.wikimedia.org/T169488#3421185 (10Marostegui) >>! In T169488#3421166, @jcrespo wrote: > ``` > root@labsdb1009[(none)... [15:32:40] 10DBA, 10Data-Services, 10Patch-For-Review, 10User-bd808, and 2 others: Drop ukwikimedia from labsdb hosts (was: ukwikimedia still present on replicas dbs on labs hosts) - https://phabricator.wikimedia.org/T169488#3421215 (10Marostegui) Revoked privileges. [15:36:37] 10DBA, 10Cloud-Services, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3421243 (10Marostegui) I have upgraded db1102 to 10.1 so we are now using rbr triggers there. After sanitizing s2 and s6, I ran the check_private... [16:23:29] 10DBA, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 5 others: Drop tables with bad data: mediawiki_page_create_1 mediawiki_revision_create_1 - https://phabricator.wikimedia.org/T169781#3421455 (10Nuria) Deployed eventlogging with fix after adding unit tests plus testing in bet... [16:59:39] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s1 - https://phabricator.wikimedia.org/T166204#3421589 (10Marostegui) db1080 is done: ``` root@neodymium:/home/marostegui# for i in `cat s1_tables`; do echo $i; mysql --skip-ssl -hdb1080 enwi... [17:00:48] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s1 - https://phabricator.wikimedia.org/T166204#3421592 (10Marostegui) [17:16:59] 10DBA, 10RESTBase-API, 10Reading List Service, 10ArchCom-RfC (ArchCom-Approved), and 4 others: RfC: Reading List service - https://phabricator.wikimedia.org/T164990#3421668 (10daniel) On July 5th, this RFC has been approved for implementation after passing the a Last Call period without any pertinent issue... [17:18:15] 10DBA, 10RESTBase-API, 10Reading List Service, 10ArchCom-RfC (ArchCom-Approved), and 4 others: RfC: Reading List service - https://phabricator.wikimedia.org/T164990#3421681 (10daniel) OUt of curiosity: what's the reasoning behind using namespace+title to refer to pages, instead of the page ID? IS this a pr... [17:51:55] 10DBA, 10RESTBase-API, 10Reading List Service, 10ArchCom-RfC (ArchCom-Approved), and 4 others: RfC: Reading List service - https://phabricator.wikimedia.org/T164990#3421869 (10Fjalapeno) @Daniel we went back and forth on this a bit. I originally proposed ids, but in the end we went with titles… basically i... [18:13:57] 10DBA, 10RESTBase-API, 10Reading List Service, 10ArchCom-RfC (ArchCom-Approved), and 4 others: RfC: Reading List service - https://phabricator.wikimedia.org/T164990#3421987 (10Tgr) ReadingLists are meant to be cross-domain (pages from multiple wikis in the same list) so the API wouldn't be able to turn pag... [18:33:37] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Reset db1070 idrac - https://phabricator.wikimedia.org/T160392#3097215 (10faidon) FYI, db1071 is in a similar state, I'm not sure why. [18:45:56] 10DBA, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 5 others: Drop tables with bad data: mediawiki_page_create_1 mediawiki_revision_create_1 - https://phabricator.wikimedia.org/T169781#3422177 (10Ottomata) Done. [19:13:26] 10DBA, 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: [Discuss] Hosting the monthly article quality dataset on labsDB - https://phabricator.wikimedia.org/T146718#2669300 (10Halfak) https://quarry.wmflabs.org/query/20169 This table is now ready for use. [19:16:27] 10DBA, 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: [Discuss] Hosting the monthly article quality dataset on labsDB - https://phabricator.wikimedia.org/T146718#3422327 (10Halfak) 05Open>03Resolved [22:32:44] 10DBA, 10Patch-For-Review: Defragment db1070, db1082, db1087, db1092 - https://phabricator.wikimedia.org/T137191#3423076 (10faidon) [22:32:47] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Reset db1070 idrac - https://phabricator.wikimedia.org/T160392#3423073 (10faidon) 05Open>03Resolved OK, so I noticed that the `Error: Unable to establish IPMI v2 / RMCP+ session` response was immediate, like the password was wrong. So I tried changi... [22:36:32] 10DBA, 10Operations: dbstore1001 troubleshoot IPMI issue - https://phabricator.wikimedia.org/T158893#3423109 (10faidon) Same issue as T160392. From the iDRAC web interface, I set the password to something random then back to our password and this seems to have done the trick. [22:36:39] 10DBA, 10Operations: dbstore1001 troubleshoot IPMI issue - https://phabricator.wikimedia.org/T158893#3423111 (10faidon) 05Open>03Resolved [22:37:01] marostegui: ^^ :)