[05:23:34] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for gomwiktionary - https://phabricator.wikimedia.org/T250706 (10Marostegui) Yeah, ETA will be more than a week unfortunately :( I will comment on this tag when ready. Thank you! [05:27:12] 10Blocked-on-schema-change, 10DBA: CentralNotice: Update DB schema on Meta for new features - https://phabricator.wikimedia.org/T254371 (10Marostegui) p:05Triage→03Medium Thanks for the ticket @AndyRussG - would you mind creating it with the format described at https://wikitech.wikimedia.org/wiki/Schema_ch... [07:25:14] 10DBA: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1091.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202006040725_marostegui_235353.log`. [07:48:05] 10DBA, 10Performance-Team, 10Performance Issue: Performance regression due to SELECT @@GLOBAL.read_only AS Value - https://phabricator.wikimedia.org/T254430 (10Nikerabbit) [07:49:12] new database consistency plan: https://xkcd.com/2315/ [07:52:03] hahaha [07:54:24] 10DBA, 10Performance-Team, 10Performance Issue: Performance regression due to SELECT @@GLOBAL.read_only AS Value - https://phabricator.wikimedia.org/T254430 (10Nikerabbit) Trying to find some causes: * 08:09 moritzm: upgrading remaining mwdebug* servers to PHP 7.2.31 [feels far fetched] * I3b4832b5a4d7410e94... [07:56:42] 10DBA: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1091.eqiad.wmnet'] ` and were **ALL** successful. [08:06:38] 10DBA, 10Performance-Team, 10Performance Issue: Performance regression due to SELECT @@GLOBAL.read_only AS Value - https://phabricator.wikimedia.org/T254430 (10Nikerabbit) [08:53:31] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) T252219 is done (only pending d... [09:11:14] 10DBA: kill_job function in remote execution module of transfer framework does not close the ports instantly - https://phabricator.wikimedia.org/T252950 (10jcrespo) 05Open→03Resolved This is resolved as defined, but more tickets may be open in the future to control concurrency and better leftover process aft... [09:11:16] 10DBA: Automate the detection of netcat listen port in transfer.py - https://phabricator.wikimedia.org/T252171 (10jcrespo) [09:13:12] 10DBA: Automate the detection of netcat listen port in transfer.py - https://phabricator.wikimedia.org/T252171 (10jcrespo) 05Open→03Resolved This is now resolved, default port it now 4400, but will open a higher port if the port is in use. --port still works, but it will fail if the port is not available.... [09:13:14] 10DBA, 10Google-Summer-of-Code (2020): GSoC 2020 Proposal: Improve the framework to transfer files over the LAN - https://phabricator.wikimedia.org/T248256 (10jcrespo) [09:14:00] 10DBA: Exception raised when setting trivial, but incorrect parameters to transfer.py - https://phabricator.wikimedia.org/T253560 (10jcrespo) 05Open→03Resolved Thanks, it works for me now, but check the comments at: https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/598562#message-e41612b356... [09:14:02] 10DBA, 10Google-Summer-of-Code (2020): GSoC 2020 Proposal: Improve the framework to transfer files over the LAN - https://phabricator.wikimedia.org/T248256 (10jcrespo) [09:15:26] jynus: should I rebase and start using transfer.py with the new additions? [09:17:03] no no [09:17:21] don't use the wmfmariadbpy version [09:17:33] ok :) [09:17:42] while all commits are stable (HEAD is stable) [09:17:58] not everthing is there to make it realease-worthy [09:18:06] gotcha [09:18:08] we may make a milestone/prerelease soon [09:18:16] for now keep using the one on path [09:18:24] we need a lot of QA still [09:18:50] also we are about to move wmfmariadbpy to a dedicated repo [09:19:01] ah ok, let me know when you want me to use it so I can help with bug hunting [09:19:12] for testing you can use it [09:19:18] just not for anything important [09:19:25] ok [09:19:49] one thing I am doing is to merge a bit more agresively now [09:19:57] just for operational reasons [09:20:04] because the repo migration [09:20:19] but I will open more tickets with some things I don't like [09:21:49] In fact, I have not yet backported the path change I did to production yet [09:22:25] hopefully we will get rid of the puppet version soon as the next task is having automated the debian package workflow [10:41:12] hashar helped me migrate transfer.py source code from wmfmariadbpy to trasnferpy repo [10:41:24] and with help me I mean "he did all the work" [10:41:38] XDDDD [10:41:57] but FYI, canonical place for transferpy is now operations/software/trasnferpy [10:42:05] but spelled right [10:42:27] I will delete transfer.py from the wmfmariadbpy but keep the repo for the rest of the scripts [10:43:24] ok! [11:06:41] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) [11:08:18] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) @daniel - testwiki has been alt... [11:28:16] 10DBA: Compress enwiki InnoDB tables - https://phabricator.wikimedia.org/T254462 (10Marostegui) [11:46:18] marostegui: to be fair I have just pushed a few buttons and copy pasted what Jynus was telling me ;) [11:46:37] team collaboration! [11:47:03] with Jenkins and Gerrit doing all the actual work \o/ [12:24:23] I've started a snapshot from db1140 into backup1002 [12:24:59] intial run seems ok, so I am going to apply the change to transfer HEAD, will wait for a proper finish to confirm backup workflow is ok on 10.4 [12:25:19] ok [12:25:51] hashar: heads up for https://gerrit.wikimedia.org/r/c/integration/config/+/602333 in this case I am required to bother you [12:39:07] yeah [12:39:09] I am troubleshooting [12:39:17] It is just the usual trigger changes [12:39:23] ok, I let you breath, let me know if I can help [12:39:32] nah, it is under control [12:39:38] I am reviewing the triggers and about to change it [12:39:49] I sometimes enjoy deep dives like the one of the recentchanges [12:40:09] and hopefully take away low prio work from you [12:40:13] this is not a deep dive it is just a trigger that needs changing because of the schema change [12:40:18] jynus: definitely helped :) [12:43:22] jynus: ;) [12:46:30] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) @daniel the new columns can be... [12:46:45] jynus: ci deployed. Should I port the patch to the new repo ? [12:47:02] let me test it, and I will ask [12:47:19] I did a recheck on your dummy change https://gerrit.wikimedia.org/r/#/c/operations/software/transferpy/+/602323/ [12:47:36] ok, we did it at the same time :-D [12:48:12] there it is, and surprisingly it worked [12:48:20] ah, becaues we haven't removed stuff yet [12:48:32] yeah, let me download the patch and apply it there, I can take care [12:48:58] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) [12:49:01] or maybe I will let him do it so I don't take authorship of the patch [12:49:36] don't worry, we will take care from here, we will just want your continuing input on the patch [12:50:11] and when ready, the ci for doc building- I am not sure if we will want it for every update, maybe just for each tag [13:00:28] jynus: git commit --amend --author "John Doe " [13:00:28] ;) [13:00:37] though yeth you would still be the owner of the change hehe [13:01:19] for the doc, we usually have ti generated on postmerge and published at something like https://doc.wikimedia.org/transferpy/ [13:01:21] yeah, I don't want to take creadit [13:01:33] for every merge? [13:01:36] and if we want to have doc maintained per tag / per branch that is doable as well [13:01:38] yeah [13:01:57] is ir normal to keep it for HEAD up to date, would that take too many resources? [13:02:05] but [13:02:06] but [13:02:11] that is running on the cloud! [13:02:19] preciselly :-D [13:02:19] it is webscale (TM) [13:02:33] I don't want to cause too much overhead [13:02:40] more seriously, there is some throttling to avoid overflowing stuff [13:02:53] docs will change a lot during gsoc, as it is one of his goals [13:02:55] and when the infra is too busy, jobs just get throttled / idle until there is some capacity [13:03:00] but may change rarely after that [13:03:05] which for doc is fine, we can afford to have ti generated with some delay [13:03:10] of course [13:03:31] and a sole volunteer is unlikely to beat the whole SRE team spamming puppet.git ;] [13:03:52] ok, in any case, you will have the last word about the config changes, so feel free to beat us with a stick [13:03:56] so yeah it can be a concern, but in the case of transferpy it is a thin drop added to the large bucket [13:04:21] yeah, the build is not precisely an openoffice.org compilation [13:04:40] it is like 10 or 20 very small text files [13:04:50] yeah ;) a minute or so probably [13:05:18] but is it intelligent, will it check that files have changed? [13:05:19] for puppet.git (which is one of the busiest repository) we have the doc generated by polling the repository instead of on each merge [13:05:51] there is no intelligence involved, that blindly runs git clone && git fetch && git checkout && tox [13:05:57] actually, it cannot be intelligent, because we use pydoc [13:06:11] so it needs to check on every commit if comments changed [13:06:14] for the doc generation that is the same, but we specify the environment to run, so same as above but with 'tox -e sphinx' [13:06:31] then there is a rsync call to transfer the doc to doc.wm.o ;] [13:06:34] I was thinking on the sphinx only part and rsts [13:06:43] yeah [13:06:59] ok, if you are not worried about that, I won't be worried :-D [13:07:05] then sphinx-apidoc generates the boilerplate for the python modules/classes so we gotta run it always [13:07:23] though one could imagine a system that prepare the code to find wether we need to rebuild the doc [13:07:36] yeah, but not worth it for this [13:07:58] indeed, easier to just blindly run everything [13:08:48] kormat: I backported the puppet change into HEAD, if you have comments: https://gerrit.wikimedia.org/r/c/operations/software/transferpy/+/602359 [14:41:41] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) [14:51:56] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) 05Stalled→03Open Blocker has been resolved. Not stalled anymore. [17:12:22] 10DBA, 10Patch-For-Review, 10Upstream, 10cloud-services-team (Kanban): Reimage labsdb1011 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T249188 (10doctaxon) Sorry for little impatience: Can you say when you are running with full capacity again? (T252209) [17:15:32] 10DBA, 10Patch-For-Review, 10Upstream, 10cloud-services-team (Kanban): Reimage labsdb1011 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T249188 (10bd808) >>! In T249188#6193558, @doctaxon wrote: > Sorry for little impatience: Can you say when you are running with full capacity again? (T252... [17:41:52] 10DBA, 10Patch-For-Review, 10Upstream, 10cloud-services-team (Kanban): Reimage labsdb1011 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T249188 (10doctaxon) > At least another week, possibly 2 more weeks. Okay super! If so it will be very helpful to make a statement about it here. [20:07:19] 10DBA, 10Gerrit: Get a writable reviewdb clone to test Gerrit upgrade with - https://phabricator.wikimedia.org/T254516 (10hashar)