[02:46:55] not sure it´s the proper channel to ask this, I want to setup wikimedia on freeBSD and I was looking for database backends : do postgreSQL is good option ? or will I be crucified on the hotel of databases ? [04:23:32] 10DBA, 10Data-Services: Consider granting `CREATE TEMPORARY TABLES` to labsdbuser - https://phabricator.wikimedia.org/T179628#3757555 (10MZMcBride) >>! In T179628#3755575, @bd808 wrote: > @MZMcBride did you have some reports that used TEMPORARY TABLE as well? I've lost track at this point of who should be invo... [04:29:40] 10DBA, 10Analytics, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3757559 (10MZMcBride) >>! In T173511#3755509, @jcrespo wrote: > As a pro- I have run many times into p... [06:18:34] !log Deploy alter table on s6 primary master (db1061) - T174569 [06:18:36] marostegui: Not expecting to hear !log here [06:18:36] T174569: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569 [06:18:41] gah [06:28:36] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3757600 (10Marostegui) db1103 is now serving s2 and s4 recentchanges services with full normal weight. [06:35:02] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop the "wb_terms.wb_terms_language" index - https://phabricator.wikimedia.org/T179106#3757602 (10Marostegui) >>! In T179106#3753731, @Marostegui wrote: > > > This is the non-compressed status: > ``` > root@db2052:~# ls -lh /srv/sqldata/wikidata... [08:37:48] elukey: for db1108 and db1107 we should remove the monitoring for s1 and s2 slave threads (check icinga for db1108) [08:37:53] as they won't replicate s1 and s2 anymore [08:39:45] marostegui: o/ - super, no idea about how to do it :D [08:39:48] is it in puppet? [08:40:18] yeah, it must be somewhere within the monitor::dba but not sure [08:40:25] jaime mentioned it yesterday, so maybe ping him [08:40:37] he wanted to do it himself i think, but maybe ping him [08:40:49] all right [08:41:13] qq - when would you be available to migrate db1046 tomorrow morning ? [08:41:22] (prepping the maintenance window email) [08:41:27] i normally start working at 7:30am XDD [08:41:31] but i rather wait for you to be online [08:41:36] so, whenever you like [08:41:51] all right, I usually start working around this hour [08:42:00] would it be ok ? [08:42:09] yeah [08:42:11] just ping me [08:42:13] super, thanks! [08:42:15] and we can start all the steps [08:58:30] not sure it´s the proper channel to ask this, I´m in the process of installing wikimedia on freeBSD for a small business and I was looking for database backends : do postgreSQL is an option that will work ? or will I run into problems that are way over my head ? [08:59:46] 10DBA, 10Data-Services: "ERROR 2006 (HY000): MySQL server has gone away" failures for a variety of queries against Wiki Replica servers - https://phabricator.wikimedia.org/T180380#3757764 (10Magnus) I will list some source below, just let me emphasize this is not limited to a specific query or type of query, p... [09:00:16] dasti_: https://www.mediawiki.org/wiki/Manual:Installation_requirements#Database_server [09:02:12] I´ve seen that, I´m looking for a ¨hand on¨ feedback [09:02:53] dasti_: Not sure how many installations with postgre are out there, in production, we only have MySQL if that helps :) [09:03:13] ok [09:03:19] So at wikimedia foundation, we use mysql [09:08:35] (I´m clearly not a specialist on that) I´ve read several times that mariadb was also the new db to use instead of mysql because of reasons/oracle ..etc...For fresh install you guys would recommend mysql ? [09:10:22] mariadb is not a new db [09:10:24] it has been there for years [09:10:28] we actually use mariadb in production [09:10:37] mariadb is a fork from mysql [09:10:54] you should have no problems with either mysql and/or mariadb, they are practically the same [09:11:04] ok [09:11:09] 10DBA, 10Data-Services: "ERROR 2006 (HY000): MySQL server has gone away" failures for a variety of queries against Wiki Replica servers - https://phabricator.wikimedia.org/T180380#3757788 (10Magnus) Update: I have added some more "re-run if failed" and "check connection/reconnect" code, and the GLAM data now s... [09:31:15] jynus: when you have a chance: https://gerrit.wikimedia.org/r/#/c/391161/ [09:41:54] 10DBA: Remove duplicate comment_temp indexes - https://phabricator.wikimedia.org/T180162#3757883 (10jcrespo) Based on my limited understanding of your description, I would do `PRIMARY (left, right)` and `UNIQUE(left)`. [09:48:32] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3757894 (10Marostegui) [09:51:35] 10DBA, 10Analytics, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3757896 (10jcrespo) > Technical contributors could certainly propose patches for this system, but an o... [10:41:34] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3758105 (10Marostegui) s6 has been done in eqiad. [10:41:44] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3758106 (10Marostegui) [10:58:31] Hi! How can I get https://www.mediawiki.org/wiki/API:Unreviewedpages by SQL. Which SQL table is this? [11:02:28] is that the flaggedrevs extension? [11:02:58] https://www.mediawiki.org/wiki/Extension:FlaggedRevs#list_%253D_unreviewedpages [11:19:14] marostegui: yes [11:27:33] Then the flaggedrevs table, wherever it is enabled [11:31:48] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop the "wb_terms.wb_terms_language" index - https://phabricator.wikimedia.org/T179106#3758274 (10Marostegui) [11:32:38] elukey: check https://gerrit.wikimedia.org/r/391182 [11:33:03] the firewall may not be active on those hosts, so you may want to check allowed hosts [11:33:23] that could be a breaking change [11:36:47] will check after lunch, thanks! [11:37:23] more like, we should setup a deploy time [11:42:11] marostegui: why is table flaggedrevs not in the actual database schema: https://upload.wikimedia.org/wikipedia/commons/9/94/MediaWiki_1.28.0_database_schema.svg [11:42:43] doctaxon: I am not in the loop, but I am going to guess that schema is for core tables only [11:42:50] extensions can create its own tables [11:42:52] doctaxon: because it is not a core table most likely [11:43:01] okay [11:43:02] doctaxon: as I said, it is an extension [11:43:06] to get a list of tables, check the information_schema_p database [11:43:49] the table schema for that particular one is: https://phabricator.wikimedia.org/P6309 [11:44:04] oh, that is helpful [12:53:04] https://gerrit.wikimedia.org/r/391198 [13:05:38] checking [13:12:00] jynus: I checked the code review and it looks fine, I am only a bit ignorant about srange => '$INTERNAL' - does it mean 10.x.x.x/whatever_subnet ips ? [13:12:36] if so it should be completely fine to merge [13:19:43] err no, https://puppet-compiler.wmflabs.org/compiler02/8757/db1046.eqiad.wmnet/change.db1046.eqiad.wmnet.err [14:05:40] 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, 10TCB-Team, and 10 others: Allow setting the watchlist table to read-only on a per-wiki basis - https://phabricator.wikimedia.org/T160062#3758870 (10Tobi_WMDE_SW) [14:06:51] elukey: https://puppet-compiler.wmflabs.org/compiler02/8759/db1046.eqiad.wmnet/ [14:07:08] https://puppet-compiler.wmflabs.org/compiler02/8759/ [14:08:39] to rebuild labsdb1009 I will be putting down labsdb1010, which should be passive right now for some hours [14:08:50] heads up [14:09:05] will you point analytics to labsdb1011 too? [14:09:12] like labsdb1011 serving both web and analytics? [14:09:25] no [14:09:35] I will not touch 9 for now [14:09:45] will only put down 10 and copy it away [14:09:54] then I will failover 9 to 10 [14:10:07] and copy data to 9 [14:10:14] aaah right, like copying 1010's data to dbstore1001 (example) and then back to 1009 [14:10:17] right right :) [14:10:29] I doubt it will fit on dbstore1001 [14:10:42] yeah, it was just an example :) [14:10:52] so something like es200 or somewhere else [14:11:48] yes, it was an example [14:19:23] 10DBA, 10Operations, 10Phabricator, 10ops-eqiad: Decommission db1048 (was Move m3 slave to db1059) - https://phabricator.wikimedia.org/T175679#3758962 (10jcrespo) @Cmjohnson sorry for the confusion- indeed **it is ok to put down db1048**. All other conversations were about failover to db1059 to substitute... [14:19:45] 10DBA, 10Operations, 10Phabricator, 10ops-eqiad: Decommission db1048 (was Move m3 slave to db1059) - https://phabricator.wikimedia.org/T175679#3758963 (10jcrespo) a:05jcrespo>03Cmjohnson [14:56:14] broken packages on labsdb1009, did any of you run upgrade on that host? [14:57:11] not me [14:58:02] what package is broken? [14:58:57] that is what I am trying to see [14:59:09] ˜/icinga-wm 15:58> RECOVERY - DPKG on labsdb1009 is OK: All packages OK [14:59:12] that just arrived [14:59:51] that is me after an apt update [15:00:19] we have no alerts right now (only db1050, decomissioned) [15:04:32] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1050 - https://phabricator.wikimedia.org/T178162#3682923 (10jcrespo) @Cmjohnson This would be on the top of the db decommissioning stack (which obviously, is not that urgent) so we can get rid of the non-useful aler... [15:05:44] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1050 - https://phabricator.wikimedia.org/T178162#3759226 (10Marostegui) And let's make sure we mark that bad disk as broken so it is not re-used somewhere else :-) [15:05:59] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3759227 (10Marostegui) db1105 has been fully pooled in s1 and s2. db1101 from s2 can now (not today, not till we are sure db1105 works fine for a day - we already have db1103 s2 and has been working fine... [15:21:29] I see, it is moritz doing some cleanups [15:25:18] 10DBA: Remove duplicate comment_temp indexes - https://phabricator.wikimedia.org/T180162#3759308 (10Anomie) 05Open>03declined Thanks. That's what we're already doing, so I'm going to decline this request to change it. [15:49:25] jynus: yeah, that was caused by my merge of https://gerrit.wikimedia.org/r/391207 [15:50:31] BTW, tomorrow I'll install openssl updates on db hosts (but only restart the common low level system services like sshd, nagios-nrpe etc. mysql will pick up the new libs whenever you're restarting mysql for DBA reasons [15:51:06] also, I'll install the new 4.9.51 kernels on jessie DB hosts, same here, new kernel can then be picked up whenever there's a reboot for hardware maintenance [16:21:37] 10DBA, 10Operations, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3759532 (10Cmjohnson) [16:21:46] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1033 and db1028 - https://phabricator.wikimedia.org/T174076#3759530 (10Cmjohnson) 05Open>03Resolved Wiped, removed from racktables [16:24:04] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1035 - https://phabricator.wikimedia.org/T176931#3759565 (10Cmjohnson) 05Open>03Resolved wiped, racktables updated. [16:24:23] 10DBA, 10Operations, 10Patch-For-Review: Decomissions old s2 eqiad hosts (db1018, db1021, db1024, db1036) - https://phabricator.wikimedia.org/T162699#3759570 (10Cmjohnson) [16:24:27] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: decommission db1036 - https://phabricator.wikimedia.org/T176311#3759568 (10Cmjohnson) 05Open>03Resolved wiped, racktables updated [16:24:45] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1037 - https://phabricator.wikimedia.org/T174902#3759573 (10Cmjohnson) 05Open>03Resolved wiped, racktables updted [16:24:48] 10DBA, 10Operations, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#2266773 (10Cmjohnson) [16:25:08] 10DBA, 10Patch-For-Review: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3759578 (10Cmjohnson) [16:25:10] 10DBA, 10Operations, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#2266774 (10Cmjohnson) [16:25:13] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1038 - https://phabricator.wikimedia.org/T177911#3759576 (10Cmjohnson) 05Open>03Resolved Wiped, racktables updated [16:25:34] 10DBA, 10Operations, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#2266776 (10Cmjohnson) [16:25:37] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1041 - https://phabricator.wikimedia.org/T173915#3759581 (10Cmjohnson) 05Open>03Resolved Wiped, racktables updated [16:59:02] > I'll install the new 4.9.51 kernels on jessie- is the package available now? or should I wait until tomorrow [18:23:28] jynus: it's already available on apt.wikimedia.org (in fact we have over 500 systems running that kernel already), you can install it with "apt-get install -y linux-meta" [18:23:51] the new kernel has a new ABI, i.e. it's a new package which is coinstallable with the previous jessie kernel [18:24:24] (linux-image-4.9.0.0.bpo.4-amd64) [20:54:00] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3760736 (10Marostegui) The following hosts are done in s5 eqiad: dbstore1002 dbstore1001 db1104 db1100 db1106