[06:07:10] 10DBA, 10Operations, 10ops-eqiad: dbproxy1012 power supply without power - https://phabricator.wikimedia.org/T217394 (10Marostegui) Thank you! ` properties CreationTimestamp = 20190307150222.000000-360 ElementName = System Event Log Entry RecordData = The input power for power supply 2 has been restor... [08:43:26] 10DBA, 10Data-Services: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127 (10Marostegui) [08:43:32] 10Blocked-on-schema-change, 10MediaWiki-Database, 10MW-1.32-notes (WMF-deploy-2018-07-17 (1.32.0-wmf.13)), 10Schema-change: Add index log_type_action - https://phabricator.wikimedia.org/T51199 (10Marostegui) [09:16:09] marostegui: see lines starting from 143 in our etherpad [09:16:41] I have organized a bit the etherpad [09:16:53] And I put the goals ideas on line 34 [09:16:59] As a general parking for ideas [09:18:18] mine is supposed to be next year's roadmap [09:18:27] sure :) [09:18:30] just saying [09:18:39] check your email for an upcoming meeting tomorrow :p [09:26:22] 10Blocked-on-schema-change, 10MediaWiki-Database, 10MW-1.32-notes (WMF-deploy-2018-07-17 (1.32.0-wmf.13)), 10Schema-change: Add index log_type_action - https://phabricator.wikimedia.org/T51199 (10Marostegui) [09:26:27] 10DBA, 10Data-Services: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127 (10Marostegui) [09:50:53] jynus: not before 2025, https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Growth#Revision_count :) just added a table [09:50:58] unless something unexpected happens [09:51:46] 2025 would happen if we saw continued growth and a pretty high rate, the other end of the scale would look more like 2030 kind of time [09:52:04] are there any other tables for wikidata that we should worry about needing to switch to bigints for? [09:56:12] addshore: I sent you a calendar invite for next tuesday's network maintenance, where we have to put the wikidictionaries on read only, just a reminder :) [09:58:01] addshore: I am not sure with MCR [09:58:24] marostegui: thanks, I'll be in berlin and in the office for it so I wont miss it! [09:58:31] thanks [09:58:45] jynus: none of the MCR tables should outgrow revisions (I had the unfortunate pleasure of working on mcr lots) [09:59:02] well, in the past we had text [09:59:10] which was more or less the same size [09:59:50] so, if wikidata used MCR, then the slots table would be an issue, but right now each page only has 1 slot [09:59:55] and then all tables that use rev_id, even if they are not as big need to be changed [10:01:32] we should really update the mysql schema diagram on mw.ogr, im not sure who normally does that [10:02:37] on commons i believe the slots table will now outgrow text or content or revisions [10:03:32] apparently MCR already has rev_id bigint on some tables [10:03:44] but revision has not [10:04:01] aaah [10:04:25] | slot_revision_id | bigint(20) unsigned | NO | PRI | NULL | | [10:04:28] indeed, thats the slots table [10:04:48] the slots table doesnt have an auto increment ID anyway [10:05:28] so, revision and text would be the main ones to watch out for [10:05:59] you may think that thinking about 2025 may be too soon [10:06:30] but 1) this is a hard problem, a simple patch would not be enough to workaround it [10:06:49] 2) worse planing has happened in the past [10:07:11] there is a table on the sys schema that automaticalle give you ids close to be emptied [10:10:41] addshore: https://phabricator.wikimedia.org/P8198 [10:10:54] oooh [10:11:43] jynus: okay, I'll take a look at those 4 for wikidata too and at least spend 5 mins thinking about them [10:11:51] recentchanges is probably the one that will fail faster [10:12:19] and the problem is that it is not just rcs or rev id, but all tables that refer those [10:12:42] oh recentchanges on enwiki is scary [10:12:48] addshore: I added wikidata below [10:13:16] addshore: https://phabricator.wikimedia.org/P8198$50 [10:17:58] jynus: indeed, recentchanges on wikidata could fill up before revisions if edit rate increases [10:18:20] and recentchanges will go moments before cu_changes goes [10:18:27] i guess [10:24:28] 2022 - 2024 for cu_changes and recentchanges auto inc field for wikidata [10:29:18] added a bit to https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Growth#recentchanges_&_cu_changes [11:29:45] https://gerrit.wikimedia.org/r/#/c/operations/dns/+/496410/ [13:31:33] marostegui: got a minute? [13:31:59] o/ [14:55:44] somebody reported [14:55:45] phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections [14:55:49] but intermittent [14:56:03] checking [14:56:21] thx [14:57:27] 10DBA, 10MediaWiki-Database, 10WikimediaEditorTasks, 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Choose DB/Cluster for WikimediaEditorTasks tables - https://phabricator.wikimedia.org/T218302 (10Mholloway) [15:01:10] mutante: there was indeed a big spike on the master at around 14:50, but other than that, there is nothing else on the graphs, if it is a peak, might not be captured on the graphs [15:02:36] marostegui: ok, thanks for checking, i did not notice the issue personally [17:24:02] jynus: you about? [17:24:10] yes [18:27:53] 10DBA, 10Operations, 10ops-codfw: rack/setup/deploy dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10Papaul) [18:28:14] 10DBA, 10Operations, 10ops-codfw: rack/setup/deploy dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10Papaul) [18:32:24] 10DBA, 10Operations, 10ops-codfw: rack/setup/deploy dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10Papaul) @jcrespo @Marostegui - The last db server in codfw is db2096. Can you please replace the "name of the host" if we are going to use something else then db209... [18:47:10] 10DBA, 10Operations, 10ops-codfw: rack/setup/deploy dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10jcrespo) @Papaul Please see my warnings at T216137#5002854 for Chris, which applys here. I had suggested to use `dbstore` for these hosts, but @Marostegui didn't agr... [18:49:34] 10DBA, 10Operations, 10ops-codfw: rack/setup/deploy dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10Marostegui) Thanks @Papaul! The rack locations are fine I think. The hostname: I think we still need to discuss them as these hosts will not be a normal database (n... [18:50:19] haha [18:50:28] :) [18:51:35] dbstore is not a bad name, and we could live with the analytics dbstore as "the strange ones" [18:52:08] as [12]**[12] will be decom [18:53:08] hey jynus, some months ago I had you set the max connection limit for user `u11106`(my user) to just 1 connection. This was so that I could use it for testing. I had the password for that user reset and I think that restored the connection limit back to 5. Could you change it once more? :) [18:55:19] 10DBA: Design the final architecture for the database binary backups - https://phabricator.wikimedia.org/T213404 (10jcrespo) https://wikitech.wikimedia.org/wiki/MariaDB/Backups is close to be a complete description of the architecture, only missing some review and the individual application documentation. [18:56:08] sorry, musikanimal you may have to give me some context [18:56:21] maybe you have a ticket or something [18:57:20] yeah I think I asked you on Phab... still trying to find the task [18:57:37] 10DBA, 10Operations, 10ops-codfw: rack/setup/deploy dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10Marostegui) @jcrespo I am still not sure if dbstore would be a good name, just because their hardware is completely different from the existing dbstoreXXXX, but on t... [18:58:37] jynus: https://phabricator.wikimedia.org/T186730 [18:58:59] 10DBA, 10Patch-For-Review: Document clearly the mariadb backup and recovery setup - https://phabricator.wikimedia.org/T205626 (10jcrespo) https://wikitech.wikimedia.org/wiki/MariaDB/Backups is close to be a complete description of the architecture, only missing some review and the individual application docume... [18:59:02] 10DBA: Design the final architecture for the database binary backups - https://phabricator.wikimedia.org/T213404 (10Marostegui) Thanks for the effort on documenting all that! I will give it a read get back to you! [18:59:07] I'll just reopen the task :) [19:00:22] 10DBA: Design the final architecture for the database binary backups - https://phabricator.wikimedia.org/T213404 (10Marostegui) As per: T213404#4980285 can we maybe close this and continue on the more specific task {T205626}? [19:00:46] 10DBA: Change user u11106 to have max 1 open connection - https://phabricator.wikimedia.org/T186730 (10MusikAnimal) 05Resolved→03Open I had the password reset for `u11106` and that apparently put the quota back at 5 connections. Could we lower it back to 1, once more? Thank you! [19:01:03] 10DBA: Design the final architecture for the database binary backups - https://phabricator.wikimedia.org/T213404 (10jcrespo) 05Open→03Resolved a:03jcrespo [19:01:11] 10DBA, 10Goal, 10Patch-For-Review: Implement database binary backups into the production infrastructure - https://phabricator.wikimedia.org/T206203 (10jcrespo) [19:01:32] 10DBA: Design the final architecture for the database binary backups - https://phabricator.wikimedia.org/T213404 (10Marostegui) Thanks! \o/ [19:03:41] 10DBA, 10Operations, 10ops-codfw: rack/setup/deploy dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10Papaul) p:05Triage→03Normal [19:06:45] 10DBA, 10Operations, 10ops-codfw: rack/setup/deploy dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10Papaul) Thanks guys. Hopefully I will get the information needed before receiving the servers on 03/22/19. [19:10:54] 10DBA: Change user u11106 to have max 1 open connection - https://phabricator.wikimedia.org/T186730 (10jcrespo) Ok, doing. [19:18:13] 10DBA: Change user u11106 to have max 1 open connection - https://phabricator.wikimedia.org/T186730 (10jcrespo) 05Open→03Resolved Done, on wikireplicas only, as above: ` root@cumin1001:~$ for host in labsdb1009 labsdb1010 labsdb1011; do mysql.py -A -h $host mysql -e "select @@hostname, max_user_connections... [19:46:38] 10DBA, 10Operations, 10Availability (MediaWiki-MultiDC), 10Core Platform Team Backlog (Watching / External), and 2 others: Make apache/maintenance hosts TLS connections to mariadb work - https://phabricator.wikimedia.org/T175672 (10mobrovac) Would the next step here be puppetising the generation/disseminat... [20:50:52] 10Blocked-on-schema-change, 10Notifications, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10Schema-change: Remove unused bundling DB fields - https://phabricator.wikimedia.org/T143763 (10Catrope) [20:51:03] 10Blocked-on-schema-change, 10Notifications, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10Schema-change: Remove event_page_namespace and event_page_title - https://phabricator.wikimedia.org/T136427 (10Catrope) [20:51:16] 10Blocked-on-schema-change, 10Growth-Team, 10Notifications, 10Patch-For-Review, 10Schema-change: Add index on event_page_id - https://phabricator.wikimedia.org/T143961 (10Catrope) [20:51:21] 10Blocked-on-schema-change, 10DBA, 10Growth-Team, 10Notifications, 10Schema-change: Remove etp_user from echo_target_page in production - https://phabricator.wikimedia.org/T217453 (10Catrope)