[00:09:30] 10DBA, 10Regression: DB spike during rollout of wmf.20 to group1 - https://phabricator.wikimedia.org/T186764#3954595 (10demon) [06:27:33] 10DBA, 10Regression: DB spike during rollout of wmf.20 to group1 - https://phabricator.wikimedia.org/T186764#3954842 (10Marostegui) p:05Unbreak!>03Normal Decreasing priority as this is not happening anymore [07:26:36] 10DBA, 10Regression: DB spike during rollout of wmf.20 to group1 - https://phabricator.wikimedia.org/T186764#3954890 (10Marostegui) @demon where did you see that lag? I have been taking a first look at graphs for a given error and I cannot see lag being graphed or any significant spike on any of the hosts for... [08:10:43] 10DBA, 10Regression: DB spike during rollout of wmf.20 to group1 - https://phabricator.wikimedia.org/T186764#3954918 (10demon) It was over on the group1 dashboard, which reports MW and HHVM errors: [[ https://logstash.wikimedia.org/app/kibana#/dashboard/group1?_g=(refreshInterval:(display:Off,pause:!f,value:0)... [08:12:46] hey guys, we should have our meeting today I think, but would it be possible to move it two hours later or so? [08:12:57] fine by me [08:13:21] :) [08:14:06] I don't think jaime is still online, will check with him once he is back [08:14:24] mark: what hour works for you? [08:15:36] 2 maybe, but can be a bit flexible around that hour [08:15:52] ok, I will ask jaime [08:15:59] Thanks! [08:26:52] 10DBA, 10Regression: DB spike during rollout of wmf.20 to group1 - https://phabricator.wikimedia.org/T186764#3954957 (10Marostegui) >>! In T186764#3954918, @demon wrote: > It was over on the group1 dashboard, which reports MW and HHVM errors > > This coincides with my deployment @ 20:36 and rollback @ 20:39.... [09:33:53] let me know if you want me to attend the meeting too ;) [09:37:00] up to you really :) [09:37:10] I guess we'll discuss HW for next year :) [10:06:21] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3955110 (10Marostegui) 05Open>03Resolved I have finished checking s3 and it also looks good. [10:07:39] 10DBA, 10Data-Services: Remove deleted wikis from wikireplicas - https://phabricator.wikimedia.org/T186685#3955114 (10Marostegui) a:03Marostegui [10:25:46] jynus mark I have moved the meeting to 3pm as I believe jynus normally has lunch around 2pm, if that doesn't work, feel free to arrange any other hour, anytime works for me :) [10:26:31] It says 4pm to me [10:26:49] are you talking UTC or CET? :D [10:27:00] yeah, I realised my calendar was in UTC (I never changed it) [10:27:21] so moved it to 15:00 our time (we are all on the same TZ), so 15:00 our time, 2pm UTC :) [10:27:24] updated the invitation [10:27:52] I wonder why my calendar was changed to UTC [10:28:50] you know, most of spain is east of london... google knows what the right TZ for you should be ;) [10:29:54] haha [10:29:54] yeah [10:30:03] portugal has a different TZ [10:30:11] we should have the same as UK yeah [10:30:34] There is always the same discussion everytime we have to change to winter/summer time [10:30:53] although I liked the late afternoon sun when I was there ;) [10:32:29] yeah, but it doesn't make sense that we have the same hour as poland for instance [10:33:45] https://en.wikipedia.org/wiki/File:Tzdiff-Europe-winter.png + https://en.wikipedia.org/wiki/File:Tzdiff-Europe-summer.png [10:35:28] the whole EU is messed XD [10:37:19] marostegui: I tried removing myself from the invite, but I probably failed to do so [10:37:28] hope I didn't mess it up :) [10:39:12] ah, let me see if I can remove you [10:40:35] paravoid: I think it is done, I don't see you as a guest for the next events [10:40:38] good bye :_( [10:42:43] yeah, seems so [10:42:45] <3 [10:58:20] 10DBA, 10Data-Services: Remove deleted wikis from wikireplicas - https://phabricator.wikimedia.org/T186685#3955161 (10Marostegui) 05Open>03Resolved ``` root@db1095[(none)]> DROP DATABASE IF EXISTS `alswikibooks`; Query OK, 65 rows affected (9.52 sec) root@db1095[(none)]> DROP DATABASE IF EXISTS `alswikiq... [11:20:51] 10DBA, 10Data-Services: labsdb1010 crashed - https://phabricator.wikimedia.org/T186579#3955200 (10jcrespo) ``` Could not execute Delete_rows_v1 event on table hywiki.geo_tags; Can't find record in 'geo_tags', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log db1095-bin.004423, end_lo... [11:22:57] 10DBA, 10Data-Services: labsdb1010 crashed - https://phabricator.wikimedia.org/T186579#3955212 (10Marostegui) >>! In T186579#3955200, @jcrespo wrote: > ``` > Could not execute Delete_rows_v1 event on table hywiki.geo_tags; Can't find record in 'geo_tags', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; t... [11:38:01] 10DBA, 10Data-Services: labsdb1010 crashed - https://phabricator.wikimedia.org/T186579#3955252 (10Marostegui) I have inserted the missing row to keep replication flowing till we rebuild this host. [13:12:35] 10DBA, 10Patch-For-Review: s5 wikidatawiki database cleanup - https://phabricator.wikimedia.org/T184599#3955414 (10Marostegui) [13:50:43] 10DBA, 10Epic: Meta ticket: Migrate multi-source database hosts to multi-instance - https://phabricator.wikimedia.org/T159423#3955466 (10elukey) We had a chat with the Research team during the offsite and they are onboard with having a multi host/instance version of dbstore1002, but we haven't still reached o... [13:51:40] 10DBA, 10Patch-For-Review: s5 wikidatawiki database cleanup - https://phabricator.wikimedia.org/T184599#3955471 (10Marostegui) dewiki tables and database has been dropped from s8 master (db1071) - tables were renamed already for a few days. [13:52:03] 10DBA, 10Patch-For-Review: s5 wikidatawiki database cleanup - https://phabricator.wikimedia.org/T184599#3955475 (10Marostegui) [14:00:48] marostegui: I think there's a problem with the external links tables on Beta cluster [14:01:07] external links being added are not added to Special:Linksearch, etc. [14:01:18] Krenair: ^^ [14:51:13] deployment-db04 [14:51:13] may be lagging behind [15:11:06] 10DBA, 10Patch-For-Review: s5 wikidatawiki database cleanup - https://phabricator.wikimedia.org/T184599#3955786 (10Marostegui) I have renamed dewiki tables on s8 hosts. Will leave them like that and drop them on Monday. [15:17:00] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions, and 2 others: Schema change to prepare for dropping archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T185128#3955799 (10Marostegui) a:03Marostegui [15:20:08] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions, and 2 others: Schema change to prepare for dropping archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T185128#3955817 (10Marostegui) [18:42:46] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3956326 (10Jdforrester-WMF) Thank you so much for the huge amount of work. [19:12:06] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: dbstore1001 crashed: Multibit ECC errors were detected on the RAID controller. - https://phabricator.wikimedia.org/T186596#3956415 (10RobH) p:05Triage>03Normal I'm setting this to normal priority in my dc-ops triaging, as it doesn't seem to be prior... [19:13:25] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: dbstore1001 crashed: Multibit ECC errors were detected on the RAID controller. - https://phabricator.wikimedia.org/T186596#3956420 (10jcrespo) p:05Normal>03High [19:23:02] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: dbstore1001 crashed: Multibit ECC errors were detected on the RAID controller. - https://phabricator.wikimedia.org/T186596#3956432 (10jcrespo) This is high for us DBAs, not high for dc-ops (but we cannot express that difference).