[05:45:50] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4247884 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db2... [06:06:16] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4247888 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2075.codfw.wmnet'] ``` and were **ALL** successful. [07:23:54] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4248014 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db2... [07:47:01] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4248025 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db2... [07:50:48] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Apply schema changes to an isolated database and examine the results - https://phabricator.wikimedia.org/T191391#4248028 (10jcrespo) [07:50:52] 10DBA, 10Patch-For-Review: reimage db2083 back into wikidata (s8) - https://phabricator.wikimedia.org/T196047#4248027 (10jcrespo) 05Open>03Resolved [07:54:12] Shouldn't this be allowing the reimage of db2059? https://gerrit.wikimedia.org/r/#/c/436730/2/modules/install_server/files/autoinstall/netboot.cfg [07:57:26] did you apply it to install2002 ? [07:57:34] I believe so [07:57:40] But maybe I forgot to run puppet there [07:57:46] but the recipe is good, isn't it? [07:57:59] then it will be waiting on you to manually partition [07:58:07] unless you give it a recipe [07:59:26] do you want to delete all content? [07:59:31] no [07:59:34] I don't want to [08:00:46] what do you get on serial console? [08:01:22] yeah, the partitions guide [08:01:49] ? [08:02:00] ˜/jynus 9:58> then it will be waiting on you to manually partition -> that [08:13:43] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4248039 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db2... [08:33:05] mmm db2059 disk predictive failure again [08:40:00] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-log-errors: Key 'rev_user_timestamp' doesn't exist in table 'revision' - https://phabricator.wikimedia.org/T196139#4248063 (10jcrespo) [08:40:27] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-log-errors: Key 'rev_user_timestamp' doesn't exist in table 'revision' - https://phabricator.wikimedia.org/T196139#4248073 (10jcrespo) p:05Triage>03Unbreak! [08:44:48] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-log-errors: Key 'rev_user_timestamp' doesn't exist in table 'revision' - https://phabricator.wikimedia.org/T196139#4248085 (10jcrespo) [08:47:20] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-log-errors: Key 'rev_user_timestamp' doesn't exist in table 'revision' - https://phabricator.wikimedia.org/T196139#4248088 (10Marostegui) I don't see that index on tables.sql for the revision table: https://github.com/wikimedia/me... [08:48:43] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-log-errors: Key 'rev_user_timestamp' doesn't exist in table 'revision' - https://phabricator.wikimedia.org/T196139#4248089 (10jcrespo) most likely a misspell of `user_timestamp`? However, `FORCE` should not be added lightly, at le... [08:52:13] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-log-errors: Key 'rev_user_timestamp' doesn't exist in table 'revision' - https://phabricator.wikimedia.org/T196139#4248090 (10jcrespo) a:03Anomie I think it is a simple misspell at https://gerrit.wikimedia.org/r/#/c/418974/ (the... [08:52:19] ^marostegui [08:53:18] yeah [08:53:31] could be as we have rev_timestamp and user_timestamp [08:53:35] but not a combination of both [08:55:45] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-log-errors: Key 'rev_user_timestamp' doesn't exist in table 'revision' - https://phabricator.wikimedia.org/T196139#4248094 (10jcrespo) Tempted to remove the unbreak now, but I prefer someone on the know to check- for all I know, t... [08:57:45] 10DBA, 10MediaWiki-Database, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-log-errors: Key 'rev_user_timestamp' doesn't exist in table 'revision' - https://phabricator.wikimedia.org/T196139#4248099 (10Marostegui) >>! In T196139#4248090, @jcrespo wrote: > I think it is a simple misspell at https://gerrit.... [09:00:33] 10DBA, 10MediaWiki-Database, 10MediaWiki-Platform-Team, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-log-errors: Key 'rev_user_timestamp' doesn't exist in table 'revision' - https://phabricator.wikimedia.org/T196139#4248101 (10jcrespo) [09:21:18] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4248146 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2059.codfw.wmnet'] ``` Of which those **FAILED**: ```... [09:25:55] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4248154 (10Marostegui) s5 eqiad progress [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1095 [] dbstore1002 [] db1116 [... [09:26:18] 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10Schema-change: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926#4248158 (10Marostegui) s5 eqiad progress [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1095 [] dbstore1002 [] db1116 [] db1124 [] d... [09:26:21] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737#4248159 (10Marostegui) s5 eqiad progress [] labsdb1009 [] labsdb1010 [] labsdb1011 [] db1095 [] dbstore1002 [] db1116 [] db1124 [] db1082 [] db... [09:26:23] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Patch-For-Review, 10Wikidata-Ministry-Of-Magic: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193#4248160 (10Marostegui) s5 eqiad progress [] labsdb1009 [] labsdb1010 [] labsdb1011 [... [09:57:04] mydumper v0.9.5 will support SSL [09:57:18] \o/ !!!! [09:57:37] https://github.com/maxbube/mydumper/commit/97438e2d0a078f3d2dd48ed992fb6a563fa5c337 [09:57:56] great news! [09:58:46] seems like a contribution from shopify [10:05:30] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4248262 (10Marostegui) cc @Anomie So, I have altered db1113 using `set SESSION SQL_MODE=strict_trans_tables;` and after... [10:05:33] jynus: maybe I misunderstood yesterday, but did you change all m2 to allow for longer keys? [10:05:39] I just broke replication :( [10:07:55] my.cnf on disk seems correct [10:10:54] only db2044 and db1065 are missing the config at runtime [10:20:58] and... thanks for fixing it :) [13:01:31] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4248603 (10dbarratt) >>! In T193449#4247505, @dmaza wrote: > Discussion here T104099 makes my argument stronger for... [13:26:12] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4248641 (10Marostegui) [13:26:29] 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10Schema-change: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926#4248642 (10Marostegui) [13:26:48] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737#4248643 (10Marostegui) [13:27:09] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Patch-For-Review, 10Wikidata-Ministry-Of-Magic: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193#4248644 (10Marostegui) [13:32:11] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4169871 (10jcrespo) I {icon heart} that you are reading and quoting my former colleague Bill Karwin. He is normally... [14:23:52] 10DBA, 10MediaWiki-Database, 10MediaWiki-Platform-Team, 10MediaWiki-extensions-FlaggedRevs, and 2 others: Key 'rev_user_timestamp' doesn't exist in table 'revision' - https://phabricator.wikimedia.org/T196139#4248862 (10Reedy) p:05Unbreak!>03High Do the DB error logs look happier now? [14:32:05] 10DBA, 10MediaWiki-Database, 10MediaWiki-Platform-Team, 10MediaWiki-extensions-FlaggedRevs, and 2 others: Key 'rev_user_timestamp' doesn't exist in table 'revision' - https://phabricator.wikimedia.org/T196139#4248880 (10jcrespo) I think so: Last error: 2018-06-01T14:07:08 https://logstash.wikimedia.org/go... [14:44:43] 10DBA, 10MediaWiki-Database, 10MediaWiki-Platform-Team, 10MediaWiki-extensions-FlaggedRevs, and 2 others: Key 'rev_user_timestamp' doesn't exist in table 'revision' - https://phabricator.wikimedia.org/T196139#4248919 (10Reedy) >>! In T196139#4248880, @jcrespo wrote: > Do you know what was the actual impact... [14:48:51] 10DBA, 10MediaWiki-Database, 10MediaWiki-Platform-Team, 10MediaWiki-extensions-FlaggedRevs, and 2 others: Key 'rev_user_timestamp' doesn't exist in table 'revision' - https://phabricator.wikimedia.org/T196139#4248930 (10Reedy) 05Open>03Resolved Better to be safe than sorry! [14:51:57] I am playing with building a wmf-mariadb103 package [14:52:25] nice [15:59:05] there is this strange behaviour on 10.3, I read about it [15:59:23] it seems start gets stuck until loading the buffer pool finishes [15:59:32] even if it is already offering a socket [15:59:52] (or maybe there is a problem with my systemd unit) [16:02:08] the buffer pool load finished, so it is definitely my systemd unit [20:46:52] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4249789 (10dmaza) >I don't expect having prepared code in advance, but a plan of the read operations to be performe...