[06:12:37] 07Blocked-on-schema-change, 10DBA, 06Multimedia, 05MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), and 3 others: Review schema changes for T125071 - Add index to image table on all wikis - https://phabricator.wikimedia.org/T160415#3174154 (10Marostegui) codfw master (db2018) is done [06:15:56] 07Blocked-on-schema-change, 10DBA, 06Multimedia, 05MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), and 3 others: Review schema changes for T125071 - Add index to image table on all wikis - https://phabricator.wikimedia.org/T160415#3174160 (10Marostegui) 05Open>03Resolved db1075 eqiad master is... [06:20:11] 07Blocked-on-schema-change, 10DBA, 05MW-1.28-release (WMF-deploy-2016-08-30_(1.28.0-wmf.17)), 05MW-1.28-release-notes, 13Patch-For-Review: Clean up revision UNIQUE indexes - https://phabricator.wikimedia.org/T142725#3174168 (10Marostegui) [06:20:14] 10DBA, 13Patch-For-Review: Unify revision table on s7 - https://phabricator.wikimedia.org/T160390#3174166 (10Marostegui) 05Open>03Resolved db1069 and labsdb1003 are also done ``` root@neodymium:/home/marostegui# for i in `cat s7_T160390`; do echo $i; mysql -P3317 --skip-ssl -hdb1069.eqiad.wmnet $i -e "show... [06:25:30] 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3174182 (10Marostegui) db1072 is done: ``` root@neodymium:/home/marostegui# mysql --skip-ssl -hdb1072 enwiki -e "show create table revision\G" *************************... [06:29:40] 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3174183 (10Marostegui) db1093 is now fixed: ``` root@neodymium:/home/marostegui# for i in frwiki jawiki ru... [07:14:42] 10DBA: run pt-tablechecksum on s5 - https://phabricator.wikimedia.org/T161294#3174207 (10Marostegui) wikidata is finished too. Most of the differences are found on the archive table (by far), some other hosts has small differences on text and on wb_terms. But the most unhealthy table looks text by a big differen... [07:20:32] 10DBA, 10ArchCom-RfC, 10MediaWiki-Database, 07RfC: Should we bump minimum supported MySQL Version? - https://phabricator.wikimedia.org/T161232#3174208 (10Marostegui) Normally, it is assumed that MariaDB 5.5 is MySQL 5.5 and MariaDB 10.0 "is" MySQL 5.6 MariaDB 10.1 is considered to be the "equivalent" of My... [07:54:00] 10DBA, 10MediaWiki-API: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3174269 (10Marostegui) [07:57:28] 10DBA, 10MediaWiki-API, 06Operations, 10Traffic: Someone is parsing all enwiki pages using the action api at a rate of ~2M pages/hour - https://phabricator.wikimedia.org/T162129#3174291 (10fgiunchedi) p:05Triage>03High I'm triaging as high since there's potential for an outage. Did the block or rate li... [08:05:39] 10DBA, 10MediaWiki-API, 06Operations, 10Traffic: Someone is parsing all enwiki pages using the action api at a rate of ~2M pages/hour - https://phabricator.wikimedia.org/T162129#3174307 (10Marostegui) Looks like he stopped two days ago: https://grafana.wikimedia.org/dashboard/db/api-summary?orgId=1&from=14... [08:24:29] 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3174340 (10Marostegui) Servers pending to alter on enwiki: eqiad: ``` db1052 - master db1067 db1065 dbstore1002 (being altered since yesterday) dbstore1001 db1069 db1... [08:27:12] 10DBA, 13Patch-For-Review: run pt-tablechecksum on s6 - https://phabricator.wikimedia.org/T160509#3174346 (10jcrespo) dbstore2001 fixed (although seeing the pattern, that one probably should be recreated). Now checking manually on selected hosts: ``` text user_properties watchlist oldimage ``` [08:38:23] volans: o/ [08:39:02] marostegui: :) [08:39:35] jynus, marostegui: another day for switchdc testing, today a bit more challenging [08:40:02] we would like to test without the dry-run a couple of db-related tasks in the codfw->eqiad configuration [08:40:09] so that technically it would be a noop [08:40:26] but some RW query will be involved, and I want to check with you it's ok [08:40:42] then, if that pass, I'd like a little step forward too, but one thing at a time ;) [08:41:20] no query is needed to run on mysql for the failover [08:41:29] only a config change [08:41:35] maybe he meant tendril? [08:41:46] I don't know [08:42:12] no I meant SET GLOBAL read_only=True/False [08:42:37] we are not going to set eqiad as read only [08:42:49] so running with the codfw TO eqiad parameters [08:43:00] it will set the CODFW masters in RO (that are already like that) [08:43:21] and verify that both codfw and eqiad masters are in RO (and that part will fail, is expected of course) [08:43:57] this is phase3, at phase 7 it will try to set the EQIAD masters in RW and verify that EQIAD is RW and codfw is RO [08:44:13] ok [08:44:36] ad yeah, it's scary :) [08:49:11] is this a test you think I can do? happy to walk you through the code and DRY-RUN output if you want [08:49:15] more eyes the better ;) [08:50:03] I said ok [08:50:21] I though it was an ok for the explanation :) [08:50:51] great! I'll do it in a couple of minutes, I'll advise here and the task will SAL [09:00:17] 10DBA: Unify revision table on s2 - https://phabricator.wikimedia.org/T162611#3174446 (10Marostegui) dbstore2001 is finished: ``` root@neodymium:/home/marostegui/git/software/dbtools# for i in `cat /home/marostegui/T162611`; do echo $i; mysql -hdbstore2001.codfw.wmnet --skip-ssl $i -e "show create table revision... [09:06:36] I have increased the codfw activity a bit: https://grafana.wikimedia.org/dashboard/db/mysql-aggregated?orgId=1&from=now-1h&to=now&var-dc=codfw%20prometheus%2Fops&var-group=core&var-shard=es1&var-shard=es2&var-shard=es3&var-role=All [09:07:09] you have? [09:23:23] 10DBA, 10Monitoring, 06Operations, 13Patch-For-Review: Create script to monitor db dumps for backups are successful (and if not, old backups are not deleted) - https://phabricator.wikimedia.org/T151999#3174470 (10fgiunchedi) >>! In T151999#2856939, @akosiaris wrote: > So, somehow we missed the error there.... [09:37:41] 10DBA, 10Cognate, 10Wikidata, 10Wikimedia-Extension-setup, and 3 others: Create SQL database and Tables for Cognate extension to be used on Wiktionaries - https://phabricator.wikimedia.org/T162252#3174522 (10Marostegui) 05Open>03Resolved a:03Marostegui I am going to close this as resolved as the core... [09:39:26] 07Blocked-on-schema-change, 10DBA, 05MW-1.28-release (WMF-deploy-2016-08-30_(1.28.0-wmf.17)), 05MW-1.28-release-notes, 13Patch-For-Review: Clean up revision UNIQUE indexes - https://phabricator.wikimedia.org/T142725#3174527 (10Marostegui) For the record we have faced this: T162774 while cleaning up some... [09:39:36] 10DBA, 10MediaWiki-API: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3174269 (10Marostegui) [09:39:38] 07Blocked-on-schema-change, 10DBA, 05MW-1.28-release (WMF-deploy-2016-08-30_(1.28.0-wmf.17)), 05MW-1.28-release-notes, 13Patch-For-Review: Clean up revision UNIQUE indexes - https://phabricator.wikimedia.org/T142725#3174529 (10Marostegui) [09:50:08] 10DBA, 10Cognate, 10Wikidata, 10Wikimedia-Extension-setup, and 3 others: Create SQL database and Tables for Cognate extension to be used on Wiktionaries - https://phabricator.wikimedia.org/T162252#3174575 (10jcrespo) I would ask Addshore to confirm by running SELECT on the empty tables from terbium/tin, et... [09:51:17] ok I will proceed with the testing, sorry for the delay, got sidetracked and also tested more small pieces to be more confident ;) [09:51:34] I also have ready a cumin-fix command in case of emergency [09:52:45] 10DBA, 06Labs, 06Operations: fstrim: Operation not supported on Labs DBs - https://phabricator.wikimedia.org/T151746#3174577 (10Marostegui) 05Open>03Resolved Closing this as that cronjob isn't there anymore on those labs hosts (which will be decommissioned soon) and it is not present on the new ones either. [09:53:06] jynus, marostegui FYI ^^^ starting now [09:53:12] volans: ok! [09:53:58] 10DBA, 10Cognate, 10Wikidata, 10Wikimedia-Extension-setup, and 3 others: Create SQL database and Tables for Cognate extension to be used on Wiktionaries - https://phabricator.wikimedia.org/T162252#3174583 (10Marostegui) >>! In T162252#3174575, @jcrespo wrote: > I would ask Addshore to confirm by running SE... [09:57:40] hhmmmm marostegui I'm guessing something like "mwscript sql.php --wiki=enwiktionary --cluster=extension1 --wikidb=cognate_wiktionary" should work? [09:58:39] addshore: I guess so, don't know the syntax of that script :) [09:58:54] I'm not actually sure I can use it to get to extension1 [09:59:09] when we ask is because those are only used by devels [09:59:11] extension1 is the name of the cluster on db-eqiad.php at least if that helps [09:59:14] so we do not know much [09:59:36] I would ask for help [09:59:49] yeh, I get issues with the script looking for the updatelog table, I'm guessing as it is a maint script. [09:59:58] if needed (but not to us) [10:02:40] jynus: now I would have a final request, if you think it's feasible. A no it's ok too ;) [10:03:12] addshore, I think you are using the wrong script, and a scary one [10:03:22] we got some errors on logstash [10:03:35] * volans waits a less busy time [10:05:11] or it is broken for extension1 [10:05:23] I would definitely ask flow people [10:05:38] jynus: will do! and then might look into more & or file a ticket [10:05:51] yeah, it is doing stuff related to flow [10:05:59] that seems like a hardcoded thing [10:06:10] and potentially problematic [10:07:33] gosh, so hard to find stuff on the logtash dashboard, it is a mess [10:07:58] addshore, note that we do not need the check we asked you [10:08:34] I am just sugesting it so that there is nothing problematic when actual data arrives [10:09:20] it was a good suggestion indeed [10:09:28] glad you did it actually [10:09:43] sometimes the grants are bad and other stuff [10:21:41] 10DBA, 07Wikimedia-Incident: Improve db backup handling - https://phabricator.wikimedia.org/T138562#3174683 (10jcrespo) [10:26:41] 10DBA: Automate dataset backup recovery - https://phabricator.wikimedia.org/T157668#3174701 (10jcrespo) [10:26:43] 10DBA, 10Monitoring, 06Operations, 13Patch-For-Review: Create script to monitor db dumps for backups are successful (and if not, old backups are not deleted) - https://phabricator.wikimedia.org/T151999#3174702 (10jcrespo) [10:26:45] 10DBA, 06Operations: Puppetize grants for mysql backups on dbstore hosts - https://phabricator.wikimedia.org/T111929#3174703 (10jcrespo) [10:26:47] 10DBA, 07Wikimedia-Incident: Improve db backup handling - https://phabricator.wikimedia.org/T138562#3174700 (10jcrespo) [10:27:39] 10DBA, 07Wikimedia-Incident: Improve regular production database backups handling - https://phabricator.wikimedia.org/T138562#2403942 (10jcrespo) p:05Low>03Normal [10:28:15] 10DBA, 07Wikimedia-Incident: Improve regular production database backups handling - https://phabricator.wikimedia.org/T138562#2403942 (10jcrespo) [10:34:10] 10DBA, 06Operations: Create less overhead on bacula jobs - https://phabricator.wikimedia.org/T162789#3174710 (10jcrespo) [10:34:32] 10DBA, 06Operations: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3174710 (10jcrespo) [10:59:35] jynus, marostegui: better time now for my last question? :) [11:00:42] 10DBA, 06Operations, 10ops-eqiad, 13Patch-For-Review: eqiad rack/setup 11 new DB servers - https://phabricator.wikimedia.org/T162233#3174778 (10Marostegui) btw @Cmjohnson db1042 can be decommissioned (it is on b2, so maybe db1099 can take its place?): https://phabricator.wikimedia.org/T149793 [11:01:04] yes [11:02:00] ok, so as I said before, it's perfectly ok to say no, we already have a good confidence that it works [11:02:54] do you think it would be possible to pick one master in codfw and manually set RO=0 just before running the stage that set it to RO=1, to ensure it actually changes it? [11:03:27] anyone in the s1-s7, es2-es3, x1 shards [11:04:50] The only issue would be "something" trying to write to it, which is unlikely, no? [11:05:11] yes, it means it would be RW for 5~10 seconds, but of course if we don't trust that noone is trying to write [11:05:28] it might cause issues (data corruption, broken replica, etc..) that I'd like to avoid :) [11:05:46] if there is something writing to it or trying to now, we would have seen that on the logs I guess [11:06:04] I mean, I am pretty sure nothing writes to it, but if there is something by chance, bye bye data consistency [11:06:35] it is pretty unlikely [11:06:56] I know that's why I'm asking and I'm asking if there is one shard that we're more confident than the others [11:06:58] so what are you trying to actually check? [11:07:30] given that the test of before have caused 0 rows affected [11:07:51] it would be a paranoid check that when actually changing the value it works too [11:08:21] can you not check it afterwards? like change it and hten do a show global bla bla to see if it is changed? [11:08:23] so if it's something we're ok, good, otherwise I can leave without this test :) [11:08:33] what do you mean? [11:08:37] the task does: [11:08:58] set dc_from RO=1 and check both DC are RO=1 (phase 3) [11:08:59] or [11:09:18] set dc_to RO=1 and check dc_from RO=1 and dc_to RO=0 (phase 7) [11:09:51] so you want to actually check that set dc_from RO=1 worked, no? [11:10:59] in this case yes [11:12:07] If there were something trying to write to them now, I guess we'd have seen that on the logs [11:13:55] I am pretty confident nothing writes to it, but if there is, it can be a big thing [11:13:59] yes, but given the potential damage, if we're at "guess" level I'll rather not do it and trust what I have tested so far [11:14:21] yes, I agree with that [11:16:13] ok, thanks! [11:16:20] sorry! :( [11:16:33] no problem at all! :D [11:16:57] to be fair, read_only is extremely reliable [11:16:57] I've tested than then mysql command line has an error returns a non-zero exit status [11:17:21] only issues that can happen is syntax errors or other external stuff [11:17:26] so I'm pretty confident everything will work as expected and tested so far [11:17:39] if it runs with noop [11:17:42] it will run [11:18:15] that's what I'm expecting, yes :) [11:18:26] the only difference is that ... mmmh let me see if I can check it [11:21:38] no, nothing in bash_history/mysql_history/mysql error log [11:21:58] so jynus the only thing that I might be not fully sure is that it actually run the query with noop [11:22:07] I agree with you that if noop runs, the real one will run fine [11:22:20] query errors are registered on performance_schema only [11:22:40] I'm not worried about errors, I would have seen those I guess [11:22:51] and you can check query runs there, too [11:23:14] last time someone ran that query [11:25:49] true, yeah my paranoia starts just because running with --skip-column-names --batch when doing set global there is no output at all [11:25:57] nope [11:25:58] I'm pretty confident it will run properly :) [11:26:07] that is why we run the checks after all [11:26:14] nope what? [11:26:52] there is no output at all [11:27:03] can you give me the full query you run? [11:27:24] I've tested it, there is no output: [11:27:32] no [11:27:42] mysql --skip-ssl --skip-column-names --batch -e "SET GLOBAL read_only=True" [11:27:48] let me verify is the exact one is run [11:27:49] give me textP [11:28:12] is is this one, verbatim [11:28:17] I do not like that true, but that is being pedantic [11:28:30] so SET GLOBAL READ_ONLY = FALSE [11:28:41] last seen 2017-04-12 09:58:50 [11:28:45] on enwiki master eqiad [11:28:54] I know, I've checked documentation and tested locally, TRUE,True,1 are all the same in boolean context [11:29:11] that is the problem, boolean context :-) [11:29:17] yes, that is correct [11:29:24] task started at 09:58:49 [11:29:26] look at puppet puppet [11:29:41] and True 'True' undef and undefined [11:29:59] do not change it, I just do not like it in general [11:30:01] I can cast to int() if you prefer [11:30:04] no no [11:30:09] that would be worse [11:30:11] int(True) = 1 in python [11:30:18] and int(False) = 0 [11:30:22] just leave it as it is [11:31:15] it will be actually more formally correct with int() I guess [11:31:16] SELECT last_seen FROM sys.statement_analysis WHERE query like 'SET%read_only%'; [11:31:33] you can execute that to check when it was last run^ [11:31:37] and how many times it was [11:31:39] nice, thanks! [11:31:52] if it is there it was succesful [11:32:07] is this active on all DBs? [11:32:07] if not, you can check queries_with_errors_or_warnings table [11:32:15] performance_schema is [11:32:23] sys may be missing somewhere [11:32:28] mmmh doens't find results on db2017 [11:32:33] ok might be that [11:37:54] sys is on db2017 [11:37:59] are you sure it is running there? [11:38:00] ok, so on codfw, while on the es shard says that the table does not exits, on some of the other [11:38:08] did not report it [11:38:20] 2017 and 2018 [11:38:32] on the others I can see it at 09:56:01 [11:38:48] maybe it is not running at all there [11:38:49] sorry and 2029 too, doesn't show it but show the one from last year [11:39:00] I'm wondering [11:39:22] given that from the logs I have: [11:39:23] Executing commands ['mysql --skip-ssl --skip-column-names --batch -e "SET GLOBAL read_only=True"'] on '10' hosts: db[2016- [11:39:26] 2019,2023,2028-2029,2033].codfw.wmnet,es[2016,2018].codfw.wmnet [11:39:51] cumin may not be running it on 17 and 18 [11:40:22] try touch /tmp/test [11:41:23] -rw-r--r-- 1 root root 0 Apr 12 11:41 test_cumin [11:41:25] created [11:43:54] p_s is not 100% reliable, it could have been discarded at times, if monitoring makes the server slow [11:44:47] 10DBA, 10Cognate, 10Wikidata, 10Wikimedia-Extension-setup, and 3 others: Create SQL database and Tables for Cognate extension to be used on Wiktionaries - https://phabricator.wikimedia.org/T162252#3174891 (10Addshore) All looks good ``` > addshore@tin:~$ mwscript sql.php --wiki=enwiktionary --cluster=exte... [11:44:54] :) [11:47:05] I will look at it later [11:47:18] I'll do some more manual test too, but first lunch [12:49:12] we can maybe use https://tools.wmflabs.org/topviews/?project=en.wikipedia.org&platform=all-access&date=2017-04-11&excludes= to prewarm those pages [12:56:09] 10DBA, 06Operations: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3175078 (10fgiunchedi) For context: fixing this would also alleviate a current problem where long-running backup jobs stall both other backup jobs and restore jobs as well (e.g... [13:21:37] 10DBA: Use tls for dump backup generation - https://phabricator.wikimedia.org/T151583#3175145 (10jcrespo) [13:21:39] 10DBA, 07Wikimedia-Incident: Improve regular production database backups handling - https://phabricator.wikimedia.org/T138562#3175146 (10jcrespo) [13:29:20] 10DBA, 10Cognate, 10Wikidata, 10Wikimedia-Extension-setup, and 3 others: Create SQL database and Tables for Cognate extension to be used on Wiktionaries - https://phabricator.wikimedia.org/T162252#3175167 (10Marostegui) Thanks for confirming! [13:31:28] 10DBA, 06Operations, 10ops-eqiad, 13Patch-For-Review: eqiad rack/setup 11 new DB servers - https://phabricator.wikimedia.org/T162233#3175169 (10Marostegui) db1019 can also go away (T147309) and it is on b1. Just mentioning it here to make sure you are aware, just in case if it is easier for you to complet... [13:37:16] 10DBA, 06Operations, 13Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3175172 (10jcrespo) [14:15:01] 10DBA, 10MediaWiki-API: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3175223 (10Anomie) The #MediaWiki-API tag is correct. {T16102} seems to be the bug that the addition was trying to fix. The situation there isn't terribly clear, T16102#182288 says it... [14:15:29] 10DBA, 10MediaWiki-API, 13Patch-For-Review: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3174269 (10Anomie) a:03Anomie [14:27:06] 10DBA, 10MediaWiki-API, 13Patch-For-Review: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3175251 (10Marostegui) Hey Anomie - thanks for the fast response, as always! So, the query forcing the index on db1088 (original table schema with the UNIQUE):... [14:47:13] 10DBA, 10Icinga, 10Monitoring, 06Operations, 13Patch-For-Review: "db1047/eventlogging_sync processes" icinga alert is flaky since at least early January - https://phabricator.wikimedia.org/T123509#3175280 (10jcrespo) 05Open>03Resolved a:03jcrespo Not happening for a long time now. Also T124307. [14:58:35] 10DBA: Run pt-table-checksum on s4 (commonswiki) - https://phabricator.wikimedia.org/T162593#3175313 (10Marostegui) This has finished, and these are the differences found: ``` db2065: archive, geo_tags, image db2058: archive, geo_tags, image db2051: archive, geo_tags, image db2019: archive, geo_tags, image db10... [15:07:17] 10DBA: Run pt-table-checksum on s1 (enwiki) - https://phabricator.wikimedia.org/T162807#3175341 (10Marostegui) [15:07:24] 10DBA: Run pt-table-checksum on s1 (enwiki) - https://phabricator.wikimedia.org/T162807#3175354 (10Marostegui) p:05Triage>03Normal [15:26:50] 10DBA, 10MediaWiki-API, 13Patch-For-Review: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3175378 (10Anomie) >>! In T162774#3175251, @Marostegui wrote: > ``` > +------+-------------+---------------+-------+---------------+---------+---------+------+--... [15:36:38] 10DBA, 10MediaWiki-API, 13Patch-For-Review: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3174269 (10jcrespo) That will be solved when we implement a version that supports better query plans for IN (5.6, not sure which mariadb version). For now, this... [15:36:55] 10DBA, 10MediaWiki-API, 13Patch-For-Review: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3175391 (10Marostegui) >>! In T162774#3175378, @Anomie wrote: >>>! In T162774#3175251, @Marostegui wrote: >> ``` >> +------+-------------+---------------+-------... [15:37:54] 10DBA, 10MediaWiki-API, 13Patch-For-Review: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3175393 (10jcrespo) This table (T162774#3175223) is **very cool**, we should put it on mediawiki.org. I will expand it with use and ignore uses. [15:41:04] 10DBA, 10MediaWiki-API, 13Patch-For-Review: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3175397 (10jcrespo) I can reproduce Anomie's issue above, and as you know this depends highly on how updated the internal stats are and the data distribution, I... [15:45:02] 10DBA, 10MediaWiki-API, 13Patch-For-Review: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3175401 (10Marostegui) >>! In T162774#3175397, @jcrespo wrote: > I can reproduce Anomie's issue above, and as you know this depends highly on how updated the int... [15:56:03] 10DBA, 06Labs, 07Regression: Tool Labs: Add skin, language, and variant to user_properties_anon - https://phabricator.wikimedia.org/T152043#3175411 (10Andrew) 05Open>03Resolved a:03Andrew ok, I applied https://gerrit.wikimedia.org/r/#/c/347854/ and then ran maintain-meta_p --all-databases --purge --deb... [16:00:56] I will leave running some warmup scripts [16:16:02] 10DBA, 10MediaWiki-API, 13Patch-For-Review: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3175453 (10Anomie) >>! In T162774#3175389, @jcrespo wrote: > For now, this is exactly the reason why I prefer IGNORE over FORCE (despite hating both). It is slig... [16:17:53] 10DBA, 10MediaWiki-API, 13Patch-For-Review: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3175457 (10jcrespo) > How so? If the index doesn't exist, IGNORE will still complain about it. > In this particular case it happens to work out because we're no... [16:22:07] 10DBA, 10MediaWiki-API, 13Patch-For-Review: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3175473 (10Anomie) You're saying it's more common to rename indexes that are used in FORCE INDEX than it is to rename indexes that are used in IGNORE INDEX? [16:39:37] 10DBA, 10MediaWiki-API, 13Patch-For-Review: FORCE INDEX on ApiQueryLinks on templatelinks and pagelinks - https://phabricator.wikimedia.org/T162774#3175532 (10jcrespo) > You're saying it's more common to rename indexes that are used in FORCE INDEX than it is to rename indexes that are used in IGNORE INDEX?... [16:41:26] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 15User-Daniel: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3175535 (10Hall1467) @jcrespo: We have decided to update the existing wbc_entity_usage table in order to allow for statement tracking (pr... [17:19:03] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 15User-Daniel: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3175774 (10jcrespo) I did not understand your last comment, is the previous patch invalid? Do you have another patch to show me? [17:25:09] 10DBA, 06Operations, 10ops-eqiad, 13Patch-For-Review: eqiad rack/setup 11 new DB servers - https://phabricator.wikimedia.org/T162233#3175790 (10Cmjohnson) Great! I will decom those 2 servers and utilize their space. [17:31:55] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 15User-Daniel: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3175838 (10jcrespo) To clarify, I have to be specially strict in this particular case because in the past, wbc_entity_usage (with the exc... [17:43:36] 10DBA, 06Operations, 13Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3175872 (10Cmjohnson) [17:43:39] 10DBA, 06Operations, 10hardware-requests, 10ops-eqiad, 13Patch-For-Review: Decommission db1042 - https://phabricator.wikimedia.org/T149793#3175870 (10Cmjohnson) 05Open>03Resolved Removed from dns, removed from rack updated racktables and resolving [17:44:13] 10DBA, 06Operations, 13Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#2266762 (10Cmjohnson) [17:47:43] 10DBA, 07Epic, 05codfw-rollout: Database maintenance scheduled while eqiad datacenter is non primary (after the DC switchover) - https://phabricator.wikimedia.org/T155099#3175885 (10jcrespo) [17:59:41] 10DBA, 05codfw-rollout: Analyze if we want to replace some masters in eqiad while it is not active - https://phabricator.wikimedia.org/T162133#3175934 (10jcrespo) [18:08:23] 10DBA, 06Operations, 10ops-eqiad, 13Patch-For-Review: eqiad rack/setup 11 new DB servers - https://phabricator.wikimedia.org/T162233#3175952 (10Cmjohnson) @marostegui Final Placement hostname rack db1096 a6 db1097 d1 db1098 b5 db1099 b2 db1100 c2 db1101 c2 (you can use db1057 slot, as db1057 can go away:... [18:42:32] 10DBA, 06Operations, 10ops-eqiad, 13Patch-For-Review: eqiad rack/setup 11 new DB servers - https://phabricator.wikimedia.org/T162233#3176045 (10Marostegui) [18:43:08] 10DBA, 06Operations, 10ops-eqiad, 13Patch-For-Review: eqiad rack/setup 11 new DB servers - https://phabricator.wikimedia.org/T162233#3156297 (10Marostegui) >>! In T162233#3175952, @Cmjohnson wrote: > @marostegui Final Placement > > hostname rack > db1096 a6 > db1097 d1 > db1098 b5 > db1099 b2 > db1100 c2... [18:47:43] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 15User-Daniel: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3176079 (10Hall1467) @jcrespo: Related to your first comment, the patch that I provided a link to (in my comment from one month ago) is n... [20:54:31] 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#3176487 (10Krinkle) [20:55:09] 10DBA, 10MediaWiki-Database, 13Patch-For-Review, 07PostgreSQL, 07Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441#2665316 (10Krinkle) [22:07:41] 10DBA, 13Patch-For-Review: run pt-tablechecksum on s6 - https://phabricator.wikimedia.org/T160509#3176722 (10jcrespo) text is almost finished, only missing codfw master and analytics/dbstores.