[03:50:24] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10Milimetric) >>! In T209031#4752361, @Bstorm wrote: > I'm aiming to write tests for this script shortly because it is too compl... [06:37:12] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) pc2009 has been pooled in. I am going to leave the weekend go by before starting the decommission proces... [06:37:24] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) [06:44:45] 10DBA, 10Cloud-Services: Prepare and check storage layer for liwikinews - https://phabricator.wikimedia.org/T205713 (10Marostegui) So what @Bstorm refers to is to create first the views` DB (xxx_p) once that is done and due to: https://jira.mariadb.org/browse/MDEV-16466 we need to add the grants manually like... [06:50:07] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Marostegui) So, speaking about 1009-1011....are you guys sure you want to reboot all the servers the same day? It wouldn't be the first time we see issues after reboots so my... [06:55:38] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Marostegui) >>! In T85757#4750588, @Banyek wrote: >>>! In T85757#4750443, @Marostegui wrote: >> Why did db2095:3316 break? > >... [06:57:37] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Marostegui) Did you just re-started replication on db2095:3316 by doing a skip transaction? [07:24:37] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) p:05Triage>03Normal Can you please let us know when this is ready to proceed? Thanks! [08:55:06] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10aborrero) >>! In T209517#4752912, @Marostegui wrote: > So, speaking about 1009-1011....are you guys sure you want to reboot all the servers the same day? > It wouldn't be the... [08:56:01] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Marostegui) >>! In T209517#4753108, @aborrero wrote: >>>! In T209517#4752912, @Marostegui wrote: >> So, speaking about 1009-1011....are you guys sure you want to reboot all th... [09:01:02] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10aborrero) a:03aborrero [09:52:29] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10jcrespo) We stopped supporting mariadb on jessie some months ago- I am not sure you will have packages to upgrade to. [09:54:00] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Marostegui) Yeah, I was talking about 1009-1011 not toolsdb :-) [10:09:25] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10jcrespo) You only checked 2 servers on each comparison- you should check all of them- it takes approximately the same amount of time, speed it up with `--step=100000` and automate the check with --... [10:38:07] there are some ongoing errors on es1 [10:38:29] since 19/9 [10:39:48] 19th Sept? [10:43:24] strange, it is on both eqiad and codfw, but I don't see it on individual servers [10:44:25] wasn't the network thing hitting one of the es servers? [10:44:31] or is it a different thing? [10:44:46] yes, but this is of mysql errors [10:44:52] so from the ones that worked [10:45:45] but connection errors? [10:46:53] I don't know [10:47:07] they are graphed toghether with other errors [10:48:15] they started the 19 at around 8 UTC [10:48:58] Nothing relevant 19th Sept 8UTC on SAL [10:49:23] yeah [10:49:26] • 08:28 jynus: stopping db1092 and db1087 in sync [10:49:26] • 07:50 godog: bump /proc/sys/net/core/rmem_default temporarily to 2MB and bounce statsd-proxy statsite-instances on graphite1004 - https://phabricator.wikimedia.org/T196484 [10:49:30] • 07:20 marostegui: Remove mwmaint1001 grants from m5 - https://phabricator.wikimedia.org/T201343 https://phabricator.wikimedia.org/T192457 [10:49:49] I was hoping for a "esXXXX" restarted/upgraded or something [10:50:33] I checked also the 18th [10:50:55] that was the day some icinga checks changed [10:51:07] mayve something related to icinga authentication? [10:51:26] changed in which way? [10:59:05] i don't know just thinking of things that could give a low rate of errors (0.3/s) [11:00:01] The last time it happened, what was it…some monitoring user, but can't remember which one [11:01:00] Ah no, last time it was db1114 and it was atop [11:01:06] https://phabricator.wikimedia.org/T191996 [12:31:55] 10DBA, 10Data-Services, 10Datasets-General-or-Unknown, 10User-notice: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10Johan) Added to: https://meta.wikimedia.org/wiki/Tech/News/2018/47 [14:30:25] 10DBA, 10User-Banyek: db1118 mysql process crashed (mysql 8.0 test host) - https://phabricator.wikimedia.org/T204594 (10Marostegui) 05Open>03declined I have needed to re-image this host to work on an important external request, so unfortunately we won't be able to debug this crash any further. Will close t... [14:47:11] 10DBA, 10User-Banyek: db1118 mysql process crashed (mysql 8.0 test host) - https://phabricator.wikimedia.org/T204594 (10jcrespo) To be fair, it was clear this was due to enabling gtid on its master, something it is clearly not supported- so no more work was possible. [15:07:26] 10DBA, 10MediaWiki-API, 10MediaWiki-Database, 10MW-1.32-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), and 2 others: ApiQueryExtLinksUsage::run query has crazy limit - https://phabricator.wikimedia.org/T59176 (10Anomie) [15:07:41] 10DBA, 10MediaWiki-API, 10MediaWiki-Database, 10MW-1.32-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), and 2 others: ApiQueryExtLinksUsage::run query has crazy limit - https://phabricator.wikimedia.org/T59176 (10Anomie) 05Open>03Resolved [15:11:16] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Bstorm) >>! In T209517#4753183, @jcrespo wrote: > We stopped supporting mariadb on jessie some months ago- I am not sure you will have packages to upgrade to. Well that makes... [15:19:33] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Anomie) >>! In T209591#4750448, @Jdforrester-WMF wrote: > (`1.33.0-wmf.5` won't ever get cut, as there's no train next week.) That's part of why I said "1.33.0-wmf.5 or later".[1] [1]: The other part is the potent... [16:17:21] 10DBA, 10Anti-Harassment, 10MediaWiki-User-management, 10wikitech.wikimedia.org, and 2 others: Fatal: Cannot block user at wikitech: Table 'labswiki.ipblocks_restrictions' doesn't exist - https://phabricator.wikimedia.org/T209674 (10Bawolff) [Making public is blocks work again] Tested blocking on wikitech... [16:17:28] 10DBA, 10Anti-Harassment, 10MediaWiki-User-management, 10wikitech.wikimedia.org, and 2 others: Fatal: Cannot block user at wikitech: Table 'labswiki.ipblocks_restrictions' doesn't exist - https://phabricator.wikimedia.org/T209674 (10Bawolff) [22:08:05] 10DBA, 10Data-Services, 10Datasets-General-or-Unknown, 10User-notice: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10greg) >>! In T174802#4753544, @Johan wrote: > Added to: https://meta.wikimedia.org/wiki/Tech/News/2018/47 So, basically the end... [22:36:02] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Bstorm) @Banyek Just looking to confirm that you will be available during the Toolsdb primary and secondary reboots as support to verify things are working correctly and help...