[01:47:58] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Ladsgroup) Okay, I queried logstash for the past seven days ([[https://logstash.wiki... [04:53:20] 10DBA: inverse_timestamp column exists in text table, it shouldn't - https://phabricator.wikimedia.org/T250063 (10Marostegui) 05Stalled→03Open a:03Marostegui I am executing this as part of T250066 [04:53:26] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Sustainability (Incident Prevention), 10WorkType-NewFunctionality: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [04:58:08] 10Blocked-on-schema-change, 10DBA: Extend echo_unread_wikis.euw_wiki - https://phabricator.wikimedia.org/T255174 (10Marostegui) p:05Triage→03Medium [05:18:11] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Marostegui) Thanks @Ladsgroup for the figures. The figures provided by Amir mean a... [07:11:30] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db2086.codfw.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20200612... [07:12:48] addshore: can I get my email added to those alerts? [07:14:57] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) [07:30:57] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2086.codfw.wmnet'] ` and were **ALL** successful. [07:41:00] 10DBA: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui) [07:41:39] 10DBA: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui) [08:02:20] marostegui: that is a good question [08:02:41] marostegui: I believe the alerts from that dashboard actually come through icinga, so, I think, you need to add your email to somewhere I dont have access to? [08:02:47] oh wait, I guess that means it is in puppet? [08:03:05] https://gerrit.wikimedia.org/g/operations/puppet/+/7a12b5503af02d8c65def01a8133ec85e5f8f141/modules/icinga/manifests/monitor/wikidata.pp#31 [08:04:50] ah I see [08:04:57] Maybe I need to add myself to that contact group [08:04:59] I will take a look [08:15:55] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) s6 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1005 [] db1140 [] db1139 [] db1131 [] db1125 [] db1113 [] db1098 [] db1096 []... [08:22:00] marostegui: I think the contact group is used for a few other alerts too [08:22:18] yeah, probably :( [08:22:22] marostegui: but I believe you can add individuals there? perhaps? or your own personal contact group? :D [08:22:27] haha [08:24:15] And yes, the alerts generalls fire for anything that impacts all sites, such as the fired yesterday for the session store issues, but also fire for some wikidata only things, often dbs, that dont get caught by other alters :) [08:24:47] yeah, I was interested on those [08:41:52] 10DBA: Compress enwiki InnoDB tables - https://phabricator.wikimedia.org/T254462 (10Marostegui) [08:42:36] 10DBA: Compress enwiki InnoDB tables - https://phabricator.wikimedia.org/T254462 (10Marostegui) [09:26:58] 10DBA, 10Gerrit, 10Patch-For-Review: Get a writable reviewdb clone to test Gerrit upgrade with - https://phabricator.wikimedia.org/T254516 (10Dzahn) After the merge above: on gerrit1002: ` - database = reviewdb - username = gerritro + database = reviewdb-test + username = gerrittest ` no change... [10:15:27] 10DBA: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui) [11:01:41] kormat: check https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/mariadb/files/check_mariadb.py$115 for the right port logic of name -> port assignment [11:05:11] that seems to do the opposite direction, but thanks, that's useful [11:06:13] yeah, s/logic/all port assignment/ [11:11:22] based on what you were compiling, no change would happen on production, as that is only function definitions [11:12:58] maybe what you wanted to do is puppet unit tests instead? [11:16:53] puppet:modules/bacula/ has some nice simple unit tests that could help you with setting up testing, if that may help [11:23:08] 10DBA, 10Patch-For-Review: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui) [11:25:26] 10DBA, 10Wiktionary: Cannot connect to database on nl.wiktionary - https://phabricator.wikimedia.org/T254987 (10Tgr) This was due to a mistake during {T253217}, FWIW. [11:37:29] jynus: cheers, i'll have a look [12:08:54] 10DBA, 10Operations, 10Sustainability (Incident Prevention): Batch db1074-db1079 hosts having BBU issues - https://phabricator.wikimedia.org/T233569 (10Marostegui) 05Open→03Declined Declining as these hosts will be refreshed next FY [12:08:57] 10DBA, 10Operations, 10ops-eqiad, 10Wikimedia-Incident: db1075 (s3 master) crashed - BBU failure - https://phabricator.wikimedia.org/T233534 (10Marostegui) [12:11:00] 10Blocked-on-schema-change, 10DBA: Extend echo_unread_wikis.euw_wiki - https://phabricator.wikimedia.org/T255174 (10Marostegui) While this can proceed on the slaves, changing the data type of the column cannot be done online and the table is big enough that will be locked for a few minutes, so this needs a mas... [12:19:52] 10Blocked-on-schema-change, 10DBA: Extend echo_unread_wikis.euw_wiki - https://phabricator.wikimedia.org/T255174 (10Marostegui) [12:57:13] kormat: the function mariadb::declare_instance shouldn't be on that namespace, that would be violation of the intended puppet standards- it should be on the profile::mariadb namespace [12:57:30] modules cannot include profile:: clases [12:58:02] ahh, i see. thanks [12:58:22] (this is my first time ever really touching puppet, so i appreciate the feedback :) [12:58:48] in theory profiles cannot import profile classes too, but I honestly not sure how to organize that [13:00:00] https://wikitech.wikimedia.org/wiki/Puppet_coding#Organization [13:02:03] yeah, I think profile::mariadb::X would be the prefered way [13:02:37] although we can move some logic to a mariadb:: namespace if it gets too complex [13:03:13] where in the tree would declare_instance.pp live? [13:03:52] modules/profile/manifests/mariadb/functions/declare_instance.pp? [13:04:15] (but i think that means it would be accessed as profile::mariadb::functions::declare_instance ...) [13:04:40] it is a function? [13:04:48] currently yes [13:05:00] so maybe modules/profile/functions/mariadb ? [13:05:01] though maybe it be a define? i don't know [13:05:13] ah [13:05:36] but fuctions that declare things? [13:05:50] this is a valid function https://gerrit.wikimedia.org/r/c/operations/puppet/+/605188/4/modules/mariadb/functions/multiinstance_mariadb_port.pp [13:05:52] :noideadog: [13:06:09] but https://gerrit.wikimedia.org/r/c/operations/puppet/+/605188/4/modules/mariadb/functions/declare_instance.pp seems like a declaration [13:06:15] as it has lateral effects [13:06:46] I don't think things should be declared in a function [13:08:20] i could maybe merge that into mariadb::instance [13:08:29] wouldn't a resource work for you? [13:08:41] i don't know :) [13:09:44] I am a bit lost, would you like to have a quick 15 minutes, no judgment chat so you tell me what you want to do and I tell you either the best way I know or who to ask? [13:09:52] that would be great, thank you [13:12:58] use the puppetisgreat codeword on meet :-D [13:28:43] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) Here are two ideas: * Should we only turn it on for projects that want it and... [13:49:31] another thing that I thought [13:49:58] that I didn't tell you, is to expose the port mapping on a hiera file [13:50:22] because we have that logic elsewhere and maybe in the future we could centalized in puppet [13:51:06] ah, neat. i'll look into that. [13:51:09] e.g. section - s1: 3306, ... And for scripts print that later into a file [13:51:51] in this case puppet I think is the right place as it normally doesn't change often [13:53:10] e.g: backup monitoring: https://phabricator.wikimedia.org/source/operations-puppet/browse/production/hieradata/role/common/mariadb/misc/zarcillo.yaml [13:56:26] used at: https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/profile/manifests/mariadb/backup/check.pp [13:58:11] great, thanks [14:01:51] one thing you may have to check, last thing I say, I promise [14:02:10] is that you have mist_multiinstance and core_multiinstance [14:02:24] *misc_ [14:02:42] not sure if those could be integrated into one with your abstractions [14:03:07] with enough extra parameters [14:04:37] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Reedy) Noting Manuel isn't saying "it's a no from the DBAs", just that the figures a... [14:58:30] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Ladsgroup) Let me add that we can start by adding failed logins, they are pretty sma... [15:25:59] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) >>! In T253802#6219382, @Ladsgroup wrote: > Let me add that we can start by ad... [15:56:11] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10ST47) >>! In T253802#6219462, @Huji wrote: > I like @Reedy's idea of only excluding... [19:21:35] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) [22:47:06] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) [22:49:41] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) [22:51:21] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji)