[07:24:34] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) [07:30:23] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) [07:31:01] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) @Andrew I have executed the alter on labswiki. Can you run it on `labstestwiki`? Thanks! [08:00:12] 10DBA, 10Data-Services, 10Operations, 10Patch-For-Review: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011 - https://phabricator.wikimedia.org/T231520 (10Marostegui) @Bstorm @bd808 I have pushed https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/534577/3/modules/openstack/files/util... [08:06:03] 10DBA, 10Data-Services, 10Operations, 10Patch-For-Review: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011 - https://phabricator.wikimedia.org/T231520 (10Marostegui) a:03Marostegui [09:27:15] let me know what you think of https://gerrit.wikimedia.org/r/c/operations/puppet/+/584552 [09:27:39] checking [09:27:45] it is either that [09:28:06] or increasing the freshness period, both would work [09:28:11] but I think I prefer this one [09:28:37] so that means Sunday, Tue, Wed and Friday [09:28:38] No? [09:28:42] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/584552/1/modules/profile/manifests/mariadb/backup/transfer.pp [09:28:58] well, +1 day [09:29:10] because they start very late in that day [09:29:44] they finish on monday, wednesday,thursday and saturday [09:30:03] yeah, I mean the start time [09:30:27] And dumps are happening on tue? [09:34:14] yes [09:34:25] well, you know, mon->tue [09:35:10] so it is either that or increase freshness check to 4 days [09:35:56] as friday -> monday is just in the 3 day border, and it alerts right now [09:40:31] +1ed [09:45:29] working on db2090? [09:45:43] yep [09:45:47] it is downtimed [09:45:49] did it alert? [09:45:50] ok, saw it long ago on SAL [09:46:08] that is why I say we should have a SAL range, not point :-P [09:46:15] haha [09:46:18] it is true [09:46:29] !log 3h X [09:46:29] jynus: Not expecting to hear !log here [09:46:42] then a nice dashboard with alerts and ongoing maintenance [11:17:42] 10DBA, 10Data-Services, 10Packaging: Upgrade wmf-pt-kill package for Debian Buster - https://phabricator.wikimedia.org/T248843 (10jcrespo) [11:18:13] 10DBA, 10Data-Services, 10Packaging: Upgrade wmf-pt-kill package for Debian Buster - https://phabricator.wikimedia.org/T248843 (10jcrespo) p:05Triage→03Medium [11:19:28] 10DBA, 10Data-Services, 10Packaging: Upgrade wmf-pt-kill package for Debian Buster - https://phabricator.wikimedia.org/T248843 (10jcrespo) [11:22:29] 10DBA, 10Data-Services, 10Packaging: Upgrade wmf-pt-kill package for Debian Buster - https://phabricator.wikimedia.org/T248843 (10jcrespo) [11:29:34] 10DBA, 10Data-Services, 10Packaging: Upgrade wmf-pt-kill package for Debian Buster - https://phabricator.wikimedia.org/T248843 (10Marostegui) Just a side note, the _current_ package does work on buster: T242702#5915709 that's why this is a "soft" blocker for labsdb host, as the upgrade can proceed and we can... [11:30:05] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) [11:37:46] 10DBA, 10Data-Services, 10Packaging: Upgrade wmf-pt-kill package for Debian Buster - https://phabricator.wikimedia.org/T248843 (10Marostegui) Not sure if it is worth packaging the new version of pt-kill, as that's the one we found that wasn't working. We are patching 2.2.20 and the current release is 3.10 bu... [13:10:20] 10DBA, 10Data-Services, 10Packaging: Upgrade wmf-pt-kill package for Debian Buster - https://phabricator.wikimedia.org/T248843 (10Marostegui) I have compiled the package on Debian Buster and installed it for testing on db1077 (buster, test-s4): ` root@db1077:/var/log/wmf-pt-kill# systemctl start wmf-pt-kill.... [13:10:27] 10DBA, 10Data-Services, 10Packaging: Upgrade wmf-pt-kill package for Debian Buster - https://phabricator.wikimedia.org/T248843 (10Marostegui) I have compiled the package on Debian Buster and installed it for testing on db1077 (buster, test-s4): ` root@db1077:/var/log/wmf-pt-kill# systemctl start wmf-pt-kill.... [13:11:07] Weird, double message [13:12:04] 10Blocked-on-schema-change, 10DBA: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 (10Marostegui) s4 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1138 [] db1125 [] db1121 [] db1103 [] db1102 [] db1097 [] db1091 [... [13:35:17] 10DBA, 10Data-Services, 10Packaging: Upgrade wmf-pt-kill package for Debian Buster - https://phabricator.wikimedia.org/T248843 (10jcrespo) I can make an attempt to update the upstream next week; at a later time, with lower priority. [13:36:48] 10DBA, 10Data-Services, 10Packaging: Upgrade wmf-pt-kill package for Debian Buster - https://phabricator.wikimedia.org/T248843 (10Marostegui) >>! In T248843#6010295, @jcrespo wrote: > I can make an attempt to update the upstream next week; at a later time, with lower priority. Sounds good thank you! The las... [13:38:10] Re: ^, I don't expect it to be fixed, but I would prefer to not lag too much behind for other bugs [13:38:41] but low prio, just maybe if fully done, do not close the ticket, just lower it to low, assign it to me [13:42:01] jynus: sure :) [13:42:19] let me check now, as I have the buster image opened [13:42:22] might be a silly thing [13:44:59] But if they attempted to patch it and didn't work...who know [13:45:01] let me see [15:05:59] ok to merge https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/583960 ? [15:14:38] Preparing to unpack wmf-pt-kill_3.1.0-1+wmf1_all.deb ... [15:14:38] Unpacking wmf-pt-kill (3.1.0-1+wmf1) over (2.2.20-1+wmf6) ... [15:14:38] Setting up wmf-pt-kill (3.1.0-1+wmf1) ... [15:14:46] root@db1077:/home/marostegui# /usr/bin/wmf-pt-kill --version [15:14:46] wmf_pt-kill 3.1.0 [15:15:02] cool, does it work? [15:15:09] as in, it starts and kills, etc. [15:15:15] Yeah, it starts [15:15:19] I am going to test now the kills [15:15:21] and all that [15:15:32] starts and stops well [15:15:35] going to check the kills [15:15:38] ha ha, in the end I have convinced you to lose the day :-P [15:15:43] yeah [15:15:47] you successfully nerd-snipped me [15:15:51] congratulations [15:16:03] but I know enought to know that if we don't do it now, we won't do it ever [15:16:30] and we can always revert... [15:16:55] add me to the patch when you are happy to see the changes upstream did [15:17:08] yeah [15:17:13] just a line [15:17:26] but creating all the deb package what was took me longer [15:17:30] all the files and all that [15:17:33] sure [15:17:51] that is why I said I didn't have time this week, and I didn't expect you to either [15:18:01] but you successfully nerdsnipped me [15:24:33] any idea on how to generate a query that stats on Execute state? [15:24:47] on db1077? [15:25:12] The kills look good, but I want to test the bug: making sure it doesn't kill "Execute" state queries as soon as they start running [15:25:40] yep [15:31:25] I have done a quick prepared statement [15:33:12] I hope it is not quick, but slow :-PPPP [15:33:24] just quick to code :-DDD [15:33:57] yeah, mostly a big select frm revision XD [15:36:37] 10DBA, 10Patch-For-Review: Fix return code handling on transfer.py error - https://phabricator.wikimedia.org/T248661 (10jcrespo) 05Open→03Resolved yay, it works now: ` $ ./transfer.py nonexistent:/tmp nonexistent:/ ERROR: The specified source path /tmp doesn't exist on nonexistent $ ./transfer.py cumin1... [15:36:45] 10DBA, 10Epic, 10Patch-For-Review: Improve regular production database backups handling - https://phabricator.wikimedia.org/T138562 (10jcrespo) [18:06:35] 10DBA, 10Data-Services, 10Operations: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011 - https://phabricator.wikimedia.org/T231520 (10bd808) >>! In T231520#6009325, @Marostegui wrote: > I have also run some queries via Quarry and I have seen them arriving correctly to labsdb1011 via the n... [18:16:10] I think I've got the package ready [18:16:19] I will send the patch tomorrow [18:16:47] This is probably the biggest nerd-snipping jynus has successfully done in months [18:16:49] Damn it!! :) [18:16:52] See you tomorrow!