[00:05:38] <twentyafterfour>	 hauskatze: you're welcome, it'll be very useful in the future 
[00:06:13] <hauskatze>	 twentyafterfour: I can see it used often when creating the mirrors of mediawiki extensions
[00:06:53] <twentyafterfour>	 I may get around to upstreaming a better ui for editing, so that you can just toggle all uris in one action from the repo edit ui
[00:07:02] <hauskatze>	 we could even host the script in a repo in phab
[00:07:22] <hauskatze>	 oh, that'd be great -or- make all uris to turn read-only once an URI is set to observe?
[00:07:27] <twentyafterfour>	 I'm going to commit it to the scripts/ directory under our phab-deployment repo
[00:07:50] <hauskatze>	 great, I'll pull it from there later so we can get upgrades easily
[00:14:45] <hauskatze>	 legoktm: hmm, does Krinkle agree with moving coverme to Gerrit? I see he's committed through Differential some days ago
[00:15:03] <hauskatze>	 I assume he is but asking just in case
[00:16:24] <Krinkle>	 I do
[00:16:30] <Krinkle>	 I'd prefer not to have to do that
[00:16:40] <Krinkle>	 hauskatze: :)
[00:16:46] <hauskatze>	 Uitgevoerd
[00:16:56] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Parsoid-PHP: We need to run tests on the PHP code in the deploy repo - https://phabricator.wikimedia.org/T239642 (10Jdforrester-WMF) 05Open→03Resolved
[00:17:36] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Community-Tech, 10Expiring-Watchlist-Items, 10TCB-Team: Create `watchlist_expiry` table in production after wmf.19 is available - https://phabricator.wikimedia.org/T244631 (10Jdforrester-WMF) p:05Triage→03Medium
[00:17:56] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.18; 2020-02-04): PHPUnit: Drop enforceTimeLimit command and related settings, because we can't use p... - https://phabricator.wikimedia.org/T243324
[00:18:11] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Growth-Team, 10MediaWiki-extensions-GettingStarted, 10User-notice: Deprecate/undeploy the GettingStarted extension - https://phabricator.wikimedia.org/T235752 (10Jdforrester-WMF) p:05Triage→03Low
[00:19:04] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Analytics, 10Event-Platform, 10Wikimedia-Extension-setup, 10Wikimedia-extension-review-queue: Deploy EventStreamConfig extension - https://phabricator.wikimedia.org/T242122 (10Jdforrester-WMF) a:03Jdforrester-WMF
[00:19:33] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Analytics, 10Event-Platform, 10Wikimedia-Extension-setup, 10Wikimedia-extension-review-queue: Deploy EventStreamConfig extension - https://phabricator.wikimedia.org/T242122 (10Jdforrester-WMF) p:05Triage→03Medium
[00:24:27] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Community-Tech, 10Expiring-Watchlist-Items, 10TCB-Team: Create `watchlist_expiry` table in production after wmf.19 is available - https://phabricator.wikimedia.org/T244631 (10Reedy) FWIW, this really doesn't need to wait for .19 to be out ;)
[00:26:10] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Community-Tech, 10Expiring-Watchlist-Items, 10TCB-Team: Create `watchlist_expiry` table in production after wmf.19 is available - https://phabricator.wikimedia.org/T244631 (10Jdforrester-WMF) One thing at a time. :-)
[00:28:35] <hauskatze>	 Repos done; sleep time
[00:50:56] <wikibugs>	 (03PS1) 10Jforrester: Zuul: Temporarily exclude FlaggedRevs from VE's dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/571396
[00:51:00] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Zuul: Temporarily exclude FlaggedRevs from VE's dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/571396 (owner: 10Jforrester)
[00:52:06] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: Temporarily exclude FlaggedRevs from VE's dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/571396 (owner: 10Jforrester)
[00:52:28] <wikibugs>	 (03PS1) 10Jforrester: Revert "Zuul: Temporarily exclude FlaggedRevs from VE's dependencies" [integration/config] - 10https://gerrit.wikimedia.org/r/571397
[00:56:10] <xSavitar>	 James_F: Some patches were already failing due to a call to a deprecated hook in FlaggedRevs, will the revert fix it?
[00:56:19] <xSavitar>	 Or maybe it's not related?
[01:05:08] <xSavitar>	 Oh James_F, found it, https://gerrit.wikimedia.org/r/c/mediawiki/extensions/FlaggedRevs/+/571392/2/backend/FlaggedRevs.php needs to be merged.
[02:07:51] <Krenair>	 Getting really fed up of '3 notifications about objects which no longer exist or which you can no longer see were discarded.'
[02:19:48] <paladox|UKInEU>	 :O twentyafterfour, you have a 34 year old car?
[08:59:10] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[09:43:54] <Daimona>	 !log mwscript extensions/AbuseFilter/maintenance/fixOldLogEntries.php --wiki=enwiki | T228655
[09:43:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[09:43:57] <stashbot>	 T228655: Dry-run fixOldLogEntries for AbuseFilter - https://phabricator.wikimedia.org/T228655
[09:44:50] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[09:46:08] <Daimona>	 !log foreachwiki extensions/AbuseFilter/maintenance/fixOldLogEntries.php | T228655
[09:46:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:05:53] <wikibugs>	 (03PS1) 10Kosta Harlan: Echo: Add experimental job for testing with Postgres [integration/config] - 10https://gerrit.wikimedia.org/r/571471
[12:56:01] <wikibugs>	 (03PS1) 10Lokal Profil: Add Karl Wettin (WMSE) [integration/config] - 10https://gerrit.wikimedia.org/r/571478 (https://phabricator.wikimedia.org/T244839)
[15:17:36] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+1] "LGTM I guess. No real groovy knowledge either however." [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/570944 (https://phabricator.wikimedia.org/T244512) (owner: 10Dduvall)
[15:21:22] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[15:43:29] <ottomata>	 i forget...how long does it take mw config changes to get synced in beta?
[15:43:34] <ottomata>	 that happens automatically, right?
[15:44:19] <Lucas_WMDE>	 I think so, yes
[15:44:26] <Lucas_WMDE>	 happens through https://integration.wikimedia.org/ci/view/Beta/job/beta-mediawiki-config-update-eqiad/ ?
[15:45:23] <Lucas_WMDE>	 hm, looks like the latest merged changes didn’t trigger a build for whatever reason
[15:45:37] <ottomata>	 ya am looking foor https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/571509
[15:45:40] <ottomata>	 for*
[15:45:55] <Lucas_WMDE>	 ah, it’s queued in Zuul
[15:46:01] <Lucas_WMDE>	 postmerge queue
[15:46:42] <Lucas_WMDE>	 if it was merged 20 minutes ago, *maybe* CI has been too busy to schedule the build so far? not sure
[15:47:09] <ottomata>	 k
[15:47:13] <ottomata>	 will wait a bi tmore then
[15:47:44] <ottomata>	 looks good i see assets-v2 in source
[15:47:54] <ottomata>	 oops wrong room for ^
[15:53:47] <greg-g>	 this'll probably get it: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/283984/
[16:05:45] <ottomata>	 don't see it on deploy01 yet
[16:10:30] <ottomata>	 greg-g:  doeos that job not work on wmf-config
[16:10:30] <ottomata>	 ?
[16:10:48] <ottomata>	 i only see it pullying in php-master?
[16:11:39] <ottomata>	 hm i think i need https://integration.wikimedia.org/ci/view/Beta/job/beta-mediawiki-config-update-eqiad/ to run
[16:12:34] <ottomata>	 oh yeah i also still see it queued in zuul
[16:13:14] <ottomata>	 Queue: operations/mediawiki-config
[16:13:19] <ottomata>	 queued for 46 minutes
[16:14:23] <James_F>	 ottomata: Yeah, sorry, all the Wikibase patches are stealing all the available resources.
[16:14:34] <ottomata>	 ah hm ok
[16:14:57] * James_F ponders kicking some things.
[16:15:00] <ottomata>	 can I do the same fetch, merge sync-file process on deploy01 ?
[16:15:02] <ottomata>	 that i do in prod
[16:15:06] <ottomata>	 just to get my stuff out there?
[16:15:07] <James_F>	 Please, no.
[16:15:10] <ottomata>	 hah ok!
[16:15:12] <James_F>	 You'll break the sync.
[16:15:19] <ottomata>	 will wait...
[16:15:47] <James_F>	 I'll do it.
[16:15:56] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[16:17:50] <James_F>	 !log Terminated two multi-day running "parsoid-pipeline-publish" jobs
[16:17:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:22:53] <James_F>	 ottomata: Manually sync failed. Digging more.
[16:27:41] <Lucas_WMDE>	 is there anything we can do in Wikibase to avoid clogging Zuul in such situations?
[16:27:51] <Lucas_WMDE>	 other than squashing chains like that into one commit? :/
[16:29:38] <James_F>	 Lucas_WMDE: Well, the big resource hog is the code health pipeline, which is still in beta.
[16:29:56] <James_F>	 But I don't think it'd be a good idea to switch that off; it's planned to be the future.
[16:30:13] <Lucas_WMDE>	 hm, ok
[16:30:15] <James_F>	 That and long chains of commits. But I appreciate the complex architecture of Wikibase means those are necessary sometimes.
[16:31:08] <James_F>	 ottomata: Should be synced now?
[16:33:22] <James_F>	 Lucas_WMDE: Don't worry about it too much. Just try not to push 20 patch chains within three hours of anyone else having to do work. :-)
[16:34:44] <Lucas_WMDE>	 ok thanks ^^
[16:35:17] <ottomata>	 yes I see it t hanks James_F !
[16:43:04] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[16:43:15] <James_F>	 Finally.
[16:43:20] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Echo: Add experimental job for testing with Postgres [integration/config] - 10https://gerrit.wikimedia.org/r/571471 (owner: 10Kosta Harlan)
[16:44:28] <wikibugs>	 (03Merged) 10jenkins-bot: Echo: Add experimental job for testing with Postgres [integration/config] - 10https://gerrit.wikimedia.org/r/571471 (owner: 10Kosta Harlan)
[16:44:56] <James_F>	 !log Zuul: [mediawiki/extensions/Echo] Add experimental Postgres job
[16:44:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:46:36] <Daimona>	 Is the codehealth job computing code coverage on its own? Or is it just my imagination?
[16:47:33] <James_F>	 Daimona: It is. The idea is to replace the coverage and test reports with it. But right now neither set are good enough to scrap the other.
[16:48:04] <Daimona>	 Ah-ah
[16:48:21] <James_F>	 So instead we burn thousands of CPU hours a day on both sets of reports. ;-)
[16:48:31] <Daimona>	 Then it could also benefit from pcov, probably... If that's possible in the codehealth config, ofc
[16:49:00] <James_F>	 Yeah. I'm not an expert on the code health pipeline tool we're using. Coverage for C is… probable?
[16:50:33] <Daimona>	 I'm not an expert either. Let me take a look if it already offers pcov as coverage driver. That would speed up things dramatically.
[16:52:16] * James_F nods.
[16:52:25] <Daimona>	 (BTW, the full log for the codehealth job is so big it almost crashes the browser)
[16:52:34] <James_F>	 Yeah. :-(
[16:52:40] <Daimona>	 (Or maybe that's because I have an average of 40 tabs open, but well...)
[16:53:17] * James_F laughs.
[16:53:19] <James_F>	 Only 40?
[16:54:24] <James_F>	 I have … 33 Firefox windows (with an average of ~4 tabs each), this Chrome window (with all 6 comms tabs in it), and Safari open where I'm testing something.
[16:55:06] <James_F>	 I expect quite a lot of my environment, I guess.
[16:55:29] <Daimona>	 Hah, that's huge. I don't think my machine can bear that (+PhpStorm and similar tools open as well).
[16:55:58] <James_F>	 I mean, I have 4 production terminal sessions and a couple of dozen local ones too. ;-)
[16:57:12] <Daimona>	 Wow, how much memory does it take?
[16:57:18] <ottomata>	 hmmm
[16:57:22] <ottomata>	 deployment-puppetmaster04
[16:57:30] <ottomata>	  /dev/vda2        19G   19G     0 100% /
[16:58:05] <ottomata>	 a lot of it is syslog, debug log, etc.
[16:58:13] <ottomata>	 those files are rotated, i'm going to remove one so I can run puppet on my client
[16:58:49] <James_F>	 Daimona: ~700MiB of memory is assigned to my terminal; Chrome has 5GiB, and Firefox has 19GiB.
[16:59:26] <James_F>	 Atom is using > 4GiB alone. Ouch.
[17:00:29] <Daimona>	 Hah, that would explain it :)
[17:00:49] <Daimona>	 4 years ago, I thought 16gb were a lot of memory. But now...
[17:01:40] <James_F>	 Yeah. I'm going to bump one of my home boxes up to 64GiB soon.
[17:02:52] <Reedy>	 My desktop at home is.. Late 2013? And has more than 16GB ram...
[17:03:03] <Daimona>	 Seems a reasonable choice. 64G and an M.2 disk is what I'm after right now.
[17:03:19] <Reedy>	 And my desktop before that (late 2011) had 24GB...
[17:03:56] <Reedy>	 It's probably overdue some amount of overhaul
[17:05:03] * greg-g looks at laptop with 16g and is mostly happy
[17:06:02] <James_F>	 Clearly I need to run a Varnish prod-like clone in VM; that only has 128GiB of RAM, so my local machine can get away with just 256GiB, right?
[17:07:17] <Reedy>	 My laptop only has 16GB though :P, and that's nearly 5 years old too
[17:07:36] <James_F>	 Reedy: Clearly we need to get you a proper one.
[17:07:53] <Reedy>	 Lack of ram isn't really a hinderance with it
[17:10:00] <wikibugs>	 (03CR) 10Jforrester: "Looks like we need to fix a few extensions' postgres support…" [integration/config] - 10https://gerrit.wikimedia.org/r/571471 (owner: 10Kosta Harlan)
[17:10:42] <Daimona>	 Back to codehealth, apparently we just use the mediawiki-coverage.sh script
[17:11:07] <Daimona>	 Which is already touched by https://gerrit.wikimedia.org/r/#/c/567938
[17:12:58] <ottomata>	 hehe i think by clearing up some space on puppetmaster04, i've unblocked puppet runs for all of beta
[17:13:10] <ottomata>	 it is pretty overloaded atm!
[17:14:20] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-puppetmaster04 is CRITICAL: CRITICAL: deployment-prep.deployment-puppetmaster04.diskspace.root.byte_percentfree (<10.00%)
[17:16:01] <Reedy>	 ottomata: You broke it
[17:16:05] <Reedy>	 ;P
[17:18:43] <ottomata>	 heh, uh huh
[17:19:23] <James_F>	 Hmm. Why did T244892 not get reported in here?
[17:19:24] <stashbot>	 T244892: PipelineLib needs a timeout - https://phabricator.wikimedia.org/T244892
[17:19:35] <James_F>	 Is wikibugs not reporting Phab task changes?
[17:19:52] <Reedy>	 I think not. I saw someone complaining elsewhere a few hours ago
[17:20:50] <James_F>	 Hmm. No-one's filed a task.
[17:21:21] <Reedy>	 It was a ping of lego in a cabal channel :P
[17:21:30] <Reedy>	 I guess mostly people don't, someone just goes and kicks the bot
[17:21:46] <James_F>	 Reedy: Want to reboot it?
[17:21:59] <James_F>	 https://www.mediawiki.org/wiki/Wikibugs#Restarting_wikibugs
[17:22:09] <James_F>	 (I'm not a member.)
[17:24:19] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-puppetmaster04 is OK: OK: All targets OK
[17:30:50] <James_F>	 Reedy: Thank you!
[17:45:19] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-puppetmaster04 is CRITICAL: CRITICAL: deployment-prep.deployment-puppetmaster04.diskspace.root.byte_percentfree (<30.00%)
[17:47:52] <shinken-wm>	 PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused
[17:47:52] <shinken-wm>	 PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused
[18:24:06] <marxarelli>	 !log 1.35.0-wmf.19 was branched at 3ac2e9ff045dc081317c9184dc83cf863fd518be for T233867
[18:24:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[18:24:08] <stashbot>	 T233867: 1.35.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T233867
[18:38:00] <wikibugs>	 10Diffusion, 10Phabricator: Improve Diffusion URIs behaviour - https://phabricator.wikimedia.org/T244903 (10MarcoAurelio)
[18:42:51] <wikibugs>	 10Diffusion, 10Phabricator: Improve Diffusion URIs behaviour - https://phabricator.wikimedia.org/T244903 (10MarcoAurelio)
[18:53:20] <wikibugs>	 10Diffusion, 10Phabricator: Reduce the number of default URIs in Diffusion - https://phabricator.wikimedia.org/T244907 (10MarcoAurelio)
[19:25:37] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T233866 (10Jdforrester-WMF) 05Open→03Resolved This is (finally) fully deployed. Declaring it done.
[19:30:51] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Scap, 10serviceops: Define a mediawiki "version" - https://phabricator.wikimedia.org/T218412 (10thcipriani)   >>! In T218412#5027492, @Dzahn wrote: > How about this: >  > We calculate the sha1 sum of each file in /srv/medi...
[19:35:26] <wikibugs>	 10MediaWiki-Codesniffer, 10Upstream: Add sniff to ensure use of spaces (not tabs) between variable and assignment operator - https://phabricator.wikimedia.org/T232265 (10Umherirrender) Another test case - https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Auth_remoteuser/+/571076/5..6/src/AuthRemoteuserSe...
[19:44:07] <legoktm>	 hi hashar
[19:44:52] <James_F>	 legoktm: Anything I can help with/
[19:44:55] <legoktm>	 hashar: do you know where in jenkins I can configure the ssh-agent-credentials? context is moving https://phabricator.wikimedia.org/T239005 to be run by CI like a few other build repos
[19:45:02] <James_F>	 Oh, right. No.
[19:45:03] <legoktm>	 James_F: same question as yesterday :p
[19:45:15] <James_F>	 legoktm: hashar's availability is the same as yesterday. :-(
[19:46:27] <hashar>	 maor or less yes
[19:46:36] * James_F hugs hashar.
[19:48:17] <hashar>	 legoktm: oh
[19:48:18] <hashar>	 SLF
[19:48:41] <hashar>	 I commented on one of the patches how it sounded terrible to spam Gerrit/CI etc with commits from an upstrea source
[19:48:54] <hashar>	 sounded way easier to have a miantneance script to update the list on a local install but well
[19:49:28] <hashar>	 anyway for ssh, generate a keypair, have the public part in ldap somehow (via wikitech I guess?)
[19:49:57] <hashar>	 then the private part can be stored in releng secret store
[19:50:27] <hashar>	 and then added to the Jenkins credential store ( https://integration.wikimedia.org/ci/credentials/ ) , requires some admin rights I guess
[19:51:08] <hashar>	 then in jjb one need to use the wrappers  'ssh-agent-credentials' there are a few example
[19:51:12] <legoktm>	 got it, thank you :)
[19:51:26] <hashar>	 as to whether all that logic should be on the CI Jenkins, that is another story
[19:51:26] <legoktm>	 yeah, I have the jjb part ready, just wasn't sure about where to put the private key
[19:51:59] <legoktm>	 I think CI > toolforge for now, and we can discuss whether to have a list committed to git in a separate thread
[19:57:29] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-greg: Phame blog for Parsoid posts - https://phabricator.wikimedia.org/T244820 (10Peachey88) >>! In T244820#5872261, @ssastry wrote: > Copy over the blog posts to the technica...
[20:46:12] <ottomata>	 heh, puppetmaster04 / full again...
[20:47:45] <ottomata>	 rm syslog.1
[20:47:47] <ottomata>	 7.7G
[20:48:56] <Reedy>	 is it spammy post upgrades?
[20:56:03] <ottomata>	 very spammy
[20:56:20] <ottomata>	 lots of errors about hiera 1.x
[20:56:48] <ottomata>	 i'm seeing about 2000 lines per second in syslog
[20:57:09] <ottomata>	 3000 per second in /var/log/debug
[20:57:12] <ottomata>	 or more
[20:57:27] <ottomata>	 same in /var/log/debug.log
[20:57:31] <ottomata>	 sorry
[20:57:34] <ottomata>	  /var/log/daemon.log
[20:59:04] <paladox|UKInEU>	 i guess the cloud need to switch to hiera 5? At least if puppet version is >=5
[21:00:21] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-puppetmaster04 is OK: OK: All targets OK
[21:03:03] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:03:12] <hauskatze>	 ehm, is phab down?
[21:03:14] <hauskatze>	 Unhandled Exception ("HTTPFutureHTTPResponseStatus")
[21:03:26] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-puppetmaster04 is CRITICAL: (Service Check Timed Out)
[21:03:28] <Reedy>	 WFM
[21:03:29] <shinken-wm>	 PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:03:30] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-parsoid10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:03:36] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-ms-fe03 is CRITICAL: (Service Check Timed Out)
[21:03:48] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-sentry01 is CRITICAL: (Service Check Timed Out)
[21:04:02] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-kafka-jumbo-2 is CRITICAL: (Service Check Timed Out)
[21:04:08] <hauskatze>	 trying to log in via OAuth giving me all sort of errors
[21:04:09] <shinken-wm>	 PROBLEM - Puppet errors on deployment-dumps-puppetmaster02 is CRITICAL: (Service Check Timed Out)
[21:04:09] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-docker-citoid01 is CRITICAL: (Service Check Timed Out)
[21:04:17] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: (Service Check Timed Out)
[21:04:18] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-acme-chief04 is CRITICAL: (Service Check Timed Out)
[21:04:20] <hauskatze>	 Unhandled Exception ("HTTPFutureCURLResponseStatus")
[21:04:23] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-aqs03 is CRITICAL: (Service Check Timed Out)
[21:04:31] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-slave-jessie-1004 is CRITICAL: (Service Check Timed Out)
[21:04:34] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-echostore01 is CRITICAL: (Service Check Timed Out)
[21:04:39] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-jobrunner03 is CRITICAL: (Service Check Timed Out)
[21:04:42] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-puppetdb03 is CRITICAL: (Service Check Timed Out)
[21:04:47] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-mcs01 is CRITICAL: (Service Check Timed Out)
[21:04:53] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-puppetmaster03 is CRITICAL: (Service Check Timed Out)
[21:05:02] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-docker-cxserver01 is CRITICAL: (Service Check Timed Out)
[21:05:05] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-poolcounter06 is CRITICAL: (Service Check Timed Out)
[21:05:20] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: (Service Check Timed Out)
[21:05:25] <shinken-wm>	 PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster04 is CRITICAL: (Service Check Timed Out)
[21:05:30] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-ms-be05 is CRITICAL: (Service Check Timed Out)
[21:05:31] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-restbase02 is CRITICAL: (Service Check Timed Out)
[21:05:31] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-memc04 is CRITICAL: (Service Check Timed Out)
[21:05:31] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-memc07 is CRITICAL: (Service Check Timed Out)
[21:05:36] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: (Service Check Timed Out)
[21:05:38] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-agent-docker-1007 is CRITICAL: (Service Check Timed Out)
[21:05:38] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-memc06 is CRITICAL: (Service Check Timed Out)
[21:05:41] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-agent-pkgbuilder-1002 is CRITICAL: (Service Check Timed Out)
[21:05:41] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-wdqs01 is CRITICAL: (Service Check Timed Out)
[21:05:48] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-agent-pkgbuilder-1001 is CRITICAL: (Service Check Timed Out)
[21:05:48] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-db06 is CRITICAL: (Service Check Timed Out)
[21:05:49] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-sca02 is CRITICAL: (Service Check Timed Out)
[21:05:54] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-trigger-01 is CRITICAL: (Service Check Timed Out)
[21:05:54] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-cache-text05 is CRITICAL: (Service Check Timed Out)
[21:05:54] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-ms-be05 is CRITICAL: (Service Check Timed Out)
[21:05:59] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-mediawiki-parsoid10 is CRITICAL: (Service Check Timed Out)
[21:06:15] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-webperf11 is CRITICAL: (Service Check Timed Out)
[21:06:47] <shinken-wm>	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:06:47] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-slave-test-jeena is CRITICAL: (Service Check Timed Out)
[21:06:49] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-urldownloader02 is CRITICAL: (Service Check Timed Out)
[21:06:52] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-db05 is CRITICAL: (Service Check Timed Out)
[21:06:53] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-agent-docker-1013 is CRITICAL: (Service Check Timed Out)
[21:06:58] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-hadoop-test-2 is CRITICAL: (Service Check Timed Out)
[21:06:59] <icinga-wm>	 PROBLEM - https://phabricator.wikimedia.org #page on phabricator.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Phabricator
[21:07:02] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-agent-docker-1009 is CRITICAL: (Service Check Timed Out)
[21:07:02] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-puppetdb02 is CRITICAL: (Service Check Timed Out)
[21:07:05] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-chromium01 is CRITICAL: (Service Check Timed Out)
[21:07:12] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-puppetdb03 is CRITICAL: (Service Check Timed Out)
[21:07:12] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-kafka-jumbo-1 is CRITICAL: (Service Check Timed Out)
[21:07:16] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-cpjobqueue is CRITICAL: (Service Check Timed Out)
[21:07:19] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-ircd is CRITICAL: (Service Check Timed Out)
[21:07:19] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-memc05 is CRITICAL: (Service Check Timed Out)
[21:07:23] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-eventstreams-1 is CRITICAL: (Service Check Timed Out)
[21:07:25] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-cumin is CRITICAL: (Service Check Timed Out)
[21:07:29] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-castor03 is CRITICAL: (Service Check Timed Out)
[21:07:29] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-ms-be06 is CRITICAL: (Service Check Timed Out)
[21:07:30] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-agent-docker-1006 is CRITICAL: (Service Check Timed Out)
[21:07:33] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-hadoop-test-3 is CRITICAL: (Service Check Timed Out)
[21:07:34] <shinken-wm>	 PROBLEM - Puppet errors on deployment-puppetmaster04 is CRITICAL: (Service Check Timed Out)
[21:07:43] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-kafka-main-1 is CRITICAL: (Service Check Timed Out)
[21:07:43] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-dumps-puppetmaster02 is CRITICAL: (Service Check Timed Out)
[21:07:46] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-agent-docker-1010 is CRITICAL: (Service Check Timed Out)
[21:07:46] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-wikifeeds01 is CRITICAL: (Service Check Timed Out)
[21:07:46] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-elastic05 is CRITICAL: (Service Check Timed Out)
[21:07:49] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-aqs01 is CRITICAL: (Service Check Timed Out)
[21:07:52] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-acme-chief03 is CRITICAL: (Service Check Timed Out)
[21:10:04] <James_F>	 hauskatze: See -operations. eqiad issues.
[21:10:22] <hauskatze>	 Yup, I'm there too; following-up there.
[21:12:43] <icinga-wm>	 RECOVERY - https://phabricator.wikimedia.org #page on phabricator.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 36940 bytes in 0.473 second response time https://wikitech.wikimedia.org/wiki/Phabricator
[21:19:04] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-chromium01 is OK: OK: All targets OK
[21:19:07] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-agent-docker-1013 is OK: OK: All targets OK
[21:19:25] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-agent-docker-1013 is OK: OK: All targets OK
[21:19:31] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-acme-chief03 is OK: OK: All targets OK
[21:19:32] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki-parsoid10 is OK: HTTP OK: HTTP/1.1 200 OK - 91391 bytes in 3.476 second response time
[21:19:32] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-wikifeeds01 is OK: OK: All targets OK
[21:19:37] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-mediawiki-parsoid10 is OK: OK: All targets OK
[21:19:39] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-agent-pkgbuilder-1002 is OK: OK: All targets OK
[21:19:40] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-puppetdb03 is OK: OK: All targets OK
[21:19:40] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-castor03 is OK: OK: All targets OK
[21:19:44] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-db06 is OK: OK: All targets OK
[21:19:51] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-urldownloader02 is OK: OK: All targets OK
[21:19:53] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-eventstreams-1 is OK: OK: All targets OK
[21:19:55] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 91363 bytes in 9.316 second response time
[21:19:58] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-ms-fe03 is OK: OK: All targets OK
[21:20:08] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-puppetdb02 is OK: OK: All targets OK
[21:20:17] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-hadoop-test-3 is OK: OK: All targets OK
[21:20:21] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-cpjobqueue is OK: OK: All targets OK
[21:20:21] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-puppetmaster03 is OK: OK: All targets OK
[21:20:38] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-restbase02 is OK: OK: deployment-prep.deployment-restbase02.diskspace._var_log.byte_percentfree (No valid datapoints found)
[21:20:46] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-wdqs01 is OK: OK: All targets OK
[21:20:48] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-ms-be05 is OK: OK: All targets OK
[21:20:55] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-jobrunner03 is OK: OK: All targets OK
[21:20:55] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-mcs01 is OK: OK: All targets OK
[21:21:07] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-ircd is OK: OK: All targets OK
[21:21:07] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-memc07 is OK: OK: All targets OK
[21:21:07] <shinken-wm>	 RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 91831 bytes in 9.379 second response time
[21:21:07] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-trigger-01 is OK: OK: integration.integration-trigger-01.diskspace._srv.byte_percentfree (No valid datapoints found)
[21:21:07] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-sca02 is OK: OK: deployment-prep.deployment-sca02.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-sca02.diskspace._srv.byte_percentfree (No valid datapoints found)
[21:21:07] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-kafka-main-1 is OK: OK: All targets OK
[21:21:09] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-slave-test-jeena is OK: OK: All targets OK
[21:21:25] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-agent-docker-1009 is OK: OK: All targets OK
[21:21:29] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-echostore01 is OK: OK: All targets OK
[21:21:34] <shinken-wm>	 PROBLEM - Puppet staleness on integration-slave-jessie-1002 is CRITICAL: (Service Check Timed Out)
[21:21:34] <shinken-wm>	 PROBLEM - Puppet errors on deployment-db05 is CRITICAL: (Service Check Timed Out)
[21:21:34] <shinken-wm>	 PROBLEM - Puppet errors on deployment-zookeeper02 is CRITICAL: (Service Check Timed Out)
[21:21:35] <shinken-wm>	 PROBLEM - Puppet errors on deployment-maps05 is CRITICAL: (Service Check Timed Out)
[21:23:10] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-cumin is OK: OK: All targets OK
[21:23:25] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-memc05 is OK: OK: All targets OK
[21:23:30] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-docker-cxserver01 is OK: OK: All targets OK
[21:23:37] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-sentry01 is OK: OK: All targets OK
[21:23:39] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-agent-docker-1010 is OK: OK: All targets OK
[21:24:09] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-kafka-jumbo-1 is OK: OK: deployment-prep.deployment-kafka-jumbo-1.diskspace._mnt_kafka.byte_percentfree (No valid datapoints found)
[21:24:12] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-kafka-jumbo-2 is OK: OK: deployment-prep.deployment-kafka-jumbo-2.diskspace._mnt_kafka.byte_percentfree (No valid datapoints found)
[21:24:15] <shinken-wm>	 RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0]
[21:24:15] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-slave-jessie-1004 is OK: OK: All targets OK
[21:24:15] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-ms-be06 is OK: OK: All targets OK
[21:24:31] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 91363 bytes in 1.952 second response time
[21:24:48] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-chromium01 is OK: OK: All targets OK
[21:25:41] <hauskatze>	 paladox|UKInEU: ping
[21:25:47] <paladox|UKInEU>	 hauskatze pong
[21:26:01] <hauskatze>	 paladox|UKInEU: can you please run a replication for me?
[21:26:06] <paladox|UKInEU>	 sure
[21:26:09] <hauskatze>	 force a replication run I mean
[21:26:14] <hauskatze>	 don't forget the --wait :D
[21:26:26] <hauskatze>	 for wikibase/vuejs-components please
[21:26:35] <hauskatze>	 just created a github mirror
[21:26:58] <paladox|UKInEU>	 doing
[21:27:05] <hauskatze>	 !log github: Created https://github.com/wikimedia/wikibase-vuejs-components | T240224
[21:27:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:27:09] <stashbot>	 T240224: Create Github and Diffusion mirrors for wikibase/vuejs-components Gerrit repository - https://phabricator.wikimedia.org/T240224
[21:27:11] <paladox|UKInEU>	 !log ssh -p 29418 paladox@gerrit.wikimedia.org replication start wikibase/vuejs-components --wait
[21:27:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:27:24] <hauskatze>	 Thanks :D
[21:27:31] <paladox|UKInEU>	 hauskatze:
[21:27:32] <paladox|UKInEU>	 Replicate wikibase/vuejs-components ref ..all.. to github.com, Succeeded! (OK)
[21:27:32] <paladox|UKInEU>	 Replication of wikibase/vuejs-components ref ..all.. completed to 2 nodes,
[21:27:44] <hauskatze>	 Yup, it's there already
[21:27:47] <hauskatze>	 Thanks again
[21:27:50] <paladox|UKInEU>	 yw :)
[21:27:56] <hauskatze>	 I don't have git bash in this pc
[21:28:37] <paladox|UKInEU>	 heh
[21:29:49] <wikibugs>	 10Diffusion, 10GitHub-Mirrors, 10Wikidata, 10User-MarcoAurelio: Create Github and Diffusion mirrors for wikibase/vuejs-components Gerrit repository - https://phabricator.wikimedia.org/T240224 (10MarcoAurelio) 05Open→03Resolved a:03MarcoAurelio Done. I've created:  * {rWBVC} * https://github.com/wikim...
[21:32:12] <shinken-wm>	 RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 51270 bytes in 9.271 second response time
[21:40:24] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Other / Uncategorized), 10Release-Engineering-Team-TODO, 10Cloud-Services, and 2 others: Horizon hiera UI: investigate data type handling - https://phabricator.wikimedia.org/T243422 (10Andrew)
[21:43:05] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Performance-Team, 10Upstream, 10phan: Constant inheritance not detected in phan, blocking merge of a patch - https://phabricator.wikimedia.org/T244633 (10aaron) I can reproduce this with master phan as well:  ` aaron@SPECTRE-GRE3FQT:~/PhpstormProjects/m...
[21:58:22] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Performance-Team, 10Upstream, 10phan: Constant inheritance not detected in phan, blocking merge of a patch - https://phabricator.wikimedia.org/T244633 (10Daimona) @aaron Were you able to extract a minimal test case from that? If not, we can try to open...
[22:00:11] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Scap, 10serviceops: Define a mediawiki "version" - https://phabricator.wikimedia.org/T218412 (10mmodell) >>! In T218412#5866447, @Dzahn wrote: > Kind of have a hard time imagining a situation where we actually want one or...
[23:25:22] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Performance-Team, 10Upstream, 10phan: Constant inheritance not detected in phan, blocking merge of a patch - https://phabricator.wikimedia.org/T244633 (10aaron) Filed as https://github.com/phan/phan/issues/3706
[23:41:18] <twentyafterfour>	 so I just disabled publishing on the phabricator repo. Upstream seem to have removed the distinction between publishing and autoclose, among other (unfortunate) simplifications
[23:45:37] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team: Update Doxygen in CI to 1.8.17 or greater - https://phabricator.wikimedia.org/T242155 (10Krinkle)
[23:46:16] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-greg: Phame blog for Parsoid posts - https://phabricator.wikimedia.org/T244820 (10greg) 05Open→03Resolved https://phabricator.wikimedia.org/phame/blog/view/18/ is created...
[23:53:22] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-greg: Phame blog for Parsoid posts - https://phabricator.wikimedia.org/T244820 (10ssastry) >>! In T244820#5874001, @Peachey88 wrote: >>>! In T244820#5872261, @ssastry wrote: >...