[00:00:23] <RoanKattouw>	 ejegg: Also please edit my phab comment to add a link to your phantom script
[00:00:34] <RoanKattouw>	 And remember to actually run it on a timer before you stop working today :)
[00:00:51] <RoanKattouw>	 Perhaps logging to a place where we can read the output
[00:01:14] <ejegg>	 Well, I'm in CST now (New Orleans) and heading to a show at 10... So being awake at 2am is probable, not sure if I'll trust myself with CN admin though!
[00:01:33] <ejegg>	 RoanKattouw: Ahh, I meant to look into a labs instance
[00:01:39] <ejegg>	 I see there's a 'monitoring' project
[00:02:54] <RoanKattouw>	 Your homedir on stat1003 could work too
[00:03:06] <RoanKattouw>	 But yeah labs is good
[00:03:10] <RoanKattouw>	 Probably better
[00:03:28] <AndyRussG>	 RoanKattouw: ejegg: Krinkle: storm of message blob cache misses around 8:02 yesterday.... https://goo.gl/QSgBjd
[00:03:43] <AndyRussG>	 (no idea if that could be relevant)
[00:04:23] <RoanKattouw>	 Hmm those are blob misses for a ton of modules
[00:04:32] <AndyRussG>	 Maybe an objectcache stampede of some sort, causing somehow corrupted data?
[00:05:57] <AndyRussG>	 Also at 8:02 today https://goo.gl/0Cz6Su
[00:08:12] <AndyRussG>	 Though actually those aren't significant peaks it seems, if u zoom out
[00:09:02] <AndyRussG>	 https://goo.gl/iGrYfP
[00:10:02] <AndyRussG>	 ejegg: so ur script is all set to run?
[00:11:00] <ejegg>	 ahh, no phantomjs up on stat servers
[00:11:13] <ejegg>	 I'll just leave my compy on and post the results in the morning
[00:11:44] <ejegg>	 relocating
[00:11:50] <AndyRussG>	 ejegg: thx!!!!
[00:15:54] <Krinkle>	 RoanKattouw: AndyRussG: getWithSetCallback() in WANObjectCache does not cache false return valus
[00:16:07] <Krinkle>	 AndyRussG: However, it seems ChoiceDataProvider currently ignores errors
[00:16:18] <Krinkle>	 return values of select() are ignored, wrapper array is created unconditionally
[00:16:19] <AndyRussG>	 Krinkle: how so?
[00:16:43] <AndyRussG>	 aaaarg
[00:16:50] <Krinkle>	 $stuff = []; $rows = $db->select( .. ); <! array|false >; foreach( $rows); if !stuff; return stuff;
[00:17:10] <Krinkle>	 should have a if $rows == false
[00:17:15] <Krinkle>	 and if so, return false too
[00:17:31] <AndyRussG>	 Krinkle: do we get a log anywhere else when a select fails?
[00:17:59] <Krinkle>	 Actually, a DBQueryError is thrown
[00:18:24] <Krinkle>	 Which would probably bubble all the way to ResourceLoader
[00:18:34] <AndyRussG>	 Mmmhmmm
[00:18:41] <Krinkle>	 (if the query fails)
[00:19:10] <AndyRussG>	 Hmmm gonna try locally
[00:23:53] <wikibugs>	 Fundraising Sprint Waiting for Godot, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Unplanned-Sprint-Work: Central Notice: possible CN issue early on December 1st UTC - https://phabricator.wikimedia.org/T152122#2843217 (Ejegg) PhantomJS script: P4559
[00:26:40] <AndyRussG>	 Krinkle: on my local setup if I trigger a DBQueryError in ChoiceDataProvider (just adding som giberish in one of the query join conditions) I don't get an error in the apache error.log
[00:27:28] <AndyRussG>	 Just some error messages in the module content
[00:28:20] <AndyRussG>	 And the error also shows in the JS console
[00:34:52] <AndyRussG>	 In my local setup that also stops JS (for example, search box autocomplete stops working). On prod, we saw evidence that JS was working fine throughout the banner gap (for example, other EL events didn't have a comparable dip)
[00:34:57] <AndyRussG>	 ejegg: RoanKattouw ^
[00:36:34] <ejegg>	 oho!
[00:37:26] <RoanKattouw>	 Hmm interesting
[00:38:03] <RoanKattouw>	 I mean, the fact that DBQueryErrors can break JS completely sounds like a bug, but it wouldn't explain this incident as you said
[00:39:40] <AndyRussG>	 yep
[00:40:04] <AndyRussG>	 I don't know really what this logstash channel shows, but at least I can say, nothing to see here: https://goo.gl/ESZxCr
[00:40:20] <AndyRussG>	 (searching on load.php because RL)
[00:41:53] <RoanKattouw>	 Ha
[00:42:05] <RoanKattouw>	 Yeah just some lock timeout query errors for ULS RL modules
[00:43:51] <AndyRussG>	 We should also log the raw string content of the choiceData module
[00:43:56] <RoanKattouw>	 ejegg: OK, so re step #2 in my phab comment, did you write a thing that dumps choiceData from load.php?
[00:43:59] <RoanKattouw>	 Ha, jinux
[00:44:02] <RoanKattouw>	 *jinx
[00:44:22] <AndyRussG>	 8p
[00:44:30] <ejegg>	 yep! I'm just using curl for that bit
[00:44:39] <AndyRussG>	 cool!
[00:44:44] <AndyRussG>	 *coorl!
[00:45:08] <RoanKattouw>	 Oh and I see P4559 already has the verison hash
[00:45:11] <ejegg>	 I've got it set up via cron on 2 machines, and I'll try to VPN one of them through the SF office
[00:45:37] <ejegg>	 though this house has slightly shaky wifi
[00:46:02] <ejegg>	 I tried running the script on one of the CI servers since I knew they had phantomjs
[00:46:12] <ejegg>	 but it never got anything back from the request
[00:46:15] <RoanKattouw>	 Hmm
[00:46:32] <ejegg>	 not sure if it's a version issue - those are at 1.9 while i'm using 2.1 locally
[00:46:33] <RoanKattouw>	 I wonder how much effort it would be to install phantomjs on a labs intsance
[00:46:47] <ejegg>	 my VPS was also balking at it, also on 1.9
[00:47:05] <RoanKattouw>	 Yeah IIRC 2.x was a major breaking change
[00:48:18] <AndyRussG>	 I can also try on a VPS
[00:48:41] <AndyRussG>	 At least only the wifi is shakey, not the house
[00:49:04] <ejegg>	 hmm, cron's having trouble with phantomjs too. wonder if i'm missing some essential env vars
[00:50:44] <ejegg>	 let's try DISPLAY
[00:53:10] <ejegg>	 yep, that did it
[00:53:14] <AndyRussG>	 :)
[00:53:56] <ejegg>	 shoot, what's up with VPN?
[00:56:57] <AndyRussG>	 Hmm for some reason I can't get to fluorine to see dberror.log (https://wikitech.wikimedia.org/wiki/Logs)
[00:57:13] <AndyRussG>	 Searching for dberror in logstash, dunno if that's getting the same
[01:01:33] <ejegg>	 phooey, no VPN on wifi or tethered via phone. That could be really inconvenient...
[01:02:20] <Krinkle>	 RoanKattouw: As soon as we mentioned potential for db exception bubbling up, I did some thinking..
[01:02:31] <AndyRussG>	 Hmm now I can reach fluorine
[01:02:37] <Krinkle>	 RoanKattouw: So, as you know, we catch errors from module content generation, and still output the other modules
[01:02:46] <RoanKattouw>	 Yeah
[01:02:51] <Krinkle>	 RoanKattouw: But, I broke this.
[01:02:55] <Krinkle>	 RoanKattouw: Not exactly
[01:02:59] <Krinkle>	 but effectively
[01:03:12] <Krinkle>	 getScript throwing is caught in a module response
[01:04:12] <Krinkle>	 However, if getScript throws and the module uses getScript in getVersionHash (via enableModuleContentVersion or manually in getDefinitionSummary), then it is not caught by startupmodule
[01:04:17] <Krinkle>	 so startupmodule's getScript will throw
[01:04:22] <Krinkle>	 which means, no startup module
[01:04:27] <Krinkle>	 just a comment about its exception
[01:04:33] <Krinkle>	 literally nothing else
[01:04:36] <Krinkle>	 no executable code
[01:04:42] <Krinkle>	 Which is expected, but bad.
[01:04:56] <Krinkle>	 https://gerrit.wikimedia.org/r/#/c/325071/1
[01:04:58] <Krinkle>	 Working on a patch now
[01:05:06] <Krinkle>	 this may not be the case here for fundraising, but just doing it anyway
[01:06:15] <Krinkle>	 RoanKattouw: See https://gerrit.wikimedia.org/r/#/c/325071/1 for details,
[01:06:58] <ejegg>	 So, if our RL module doesn't explicitly implement a version hash, it'll use getScript
[01:10:00] <AndyRussG>	 andyrussg@fluorine:~$ gunzip -c /a/mw-log/archive/DBQuery.log-20161201.gz | grep -i choicedata
[01:10:03] <Krinkle>	 ejegg: No, that's not the default.
[01:10:04] <AndyRussG>	 ^ no results
[01:10:05] <Krinkle>	 ejegg: https://github.com/wikimedia/mediawiki-extensions-CentralNotice/blob/master/includes/CNChoiceDataResourceLoaderModule.php#L168
[01:10:24] <Krinkle>	 CN data module getModifiedHash is used - which calls getChoices - which may throw
[01:10:25] <ejegg>	 ah right
[01:10:27] <Krinkle>	 this is an RL issue
[01:10:34] <Krinkle>	 it shoudl be caught there, don't worry
[01:10:52] <Krinkle>	 getChoices is effectively getScript almost
[01:11:26] <Krinkle>	 AndyRussG: Good, okay, so it wasn't a db error
[01:11:33] <AndyRussG>	 Hmmmm
[01:12:10] <AndyRussG>	 Krinkle: so, we needn't be working ASAP on a patch to handle DB errors on those selects?
[01:13:58] <RoanKattouw>	 Krinkle: Ouch
[01:14:25] <RoanKattouw>	 No startup module is very bad
[01:15:40] <AndyRussG>	 ^ So it could be instead an error in the generation of some other module?
[01:16:00] <AndyRussG>	 How would the phantomjs script log the presence of a startup module?
[01:17:29] <AndyRussG>	 Also, for the CN issue, how might that be reflected in the pattern of no-banner-served that we see?
[01:17:48] <AndyRussG>	 I guess it might fit, since choicedata varies on lang
[01:18:57] <AndyRussG>	 Or rather in general RL varies on lang
[01:19:21] <AndyRussG>	 So I guess it could have been an issue for certain langs at certain DCs
[01:20:13] <AndyRussG>	 Krinkle: looking now at resourceloader.log on fluorine
[01:20:30] <AndyRussG>	 (dunno exactly what that'd log)
[01:21:36] <dstrine>	 I'm going to run. I'll be checking emails.
[01:23:33] <Krinkle>	 AndyRussG: Hard to detect. Probably by either looking in the response for /modules=startup/ for an exception, or by scripting a RLQ.push function and noticing that it is never called
[01:23:55] <Krinkle>	 AndyRussG: Which btw, is most likely how you'll want to execute scripts  in phantom given that mw is loaded asynchronously.
[01:24:14] <Krinkle>	 RLQ.push() will run immediately if mw is loaded, and otherwise queued for once async startup arrives
[01:24:53] <Krinkle>	 evalScriptAsync(function (callback) { RLQ.push(function () { .... callback(..); });  });
[01:24:57] <Krinkle>	 or something like that
[01:24:58] <AndyRussG>	 Krinkle: ejegg: ^ anyway a RL error in some other module doesn't explain our hour-long outage
[01:25:04] <AndyRussG>	 RoanKattouw: ^
[01:25:19] <ejegg>	 hmm
[01:25:26] <Krinkle>	 Indeed. If a random other module throw an exception once, and not again that would cause upto 5min of issues.
[01:27:11] <AndyRussG>	 Mmmm right tho it could throw repeatedly for an hour
[01:29:07] <Krinkle>	 Either way, it'd show up in exception.log
[01:30:41] <AndyRussG>	 Hmmm
[01:30:59] <AndyRussG>	 Does the objectcache ever get corrupted? Does that log somewhere?
[01:31:59] <Krinkle>	 I don't know of any objectcache exception, but if it does throw somehow, it'd bubble to RL and be in exception.og
[01:32:17] <Krinkle>	 if not, then it would return flase, bypass cahce and call fetchChoices right away from the callback
[01:32:22] <Krinkle>	 graceful fallback
[01:32:33] <Krinkle>	 also check meidawiki-errors for non-exceptions
[01:33:16] <AndyRussG>	 $ gunzip -c /a/mw-log/archive/memcached.log-20161201.gz | grep CentralNoticeChoiceData also gives naught
[01:34:31] <Krinkle>	 https://logstash.wikimedia.org/goto/3086b68df3c1c4ea4e742d1fff1fbfa7
[01:34:43] <Krinkle>	 mediawiki-error 08-09 AM url:"load.php"
[01:34:48] <Krinkle>	 08-10 AM
[01:35:56] <Krinkle>	 https://logstash.wikimedia.org/goto/4a049e04aff7c52e0aa231fc3e5a56f2 - without DBReplication warnings
[01:36:33] <Krinkle>	 https://logstash.wikimedia.org/goto/f06926e432504d73bcaa259a6fd94af3 - without wikitech/silver
[01:36:40] <Krinkle>	 leaves 4 database connect exceptions
[01:37:21] <Krinkle>	 for 4 random page views
[01:37:39] <AndyRussG>	 Hmm I don't see it...
[01:38:19] <Krinkle>	 scroll down to event list
[01:39:47] <Krinkle>	 2 for load.php on itwiki and plwiki containing centralnotice among others (would've returned the error in a comment, other modules unharmed, and cached for 5 minutes instead of 30 days)
[01:39:55] <Krinkle>	 and 2 for for load.php that doesn't contain centralnotice
[01:40:00] <AndyRussG>	 Don't see stuff at 8:09 or 8:10
[01:40:04] <Krinkle>	 indeed
[01:40:11] <Krinkle>	 these were about 20-40 minutes earlier
[01:40:46] <AndyRussG>	 Hmmm
[01:42:15] <AndyRussG>	 So... that means...? Maybe another connection error when generating choicedata that we're not detecting because not logged?
[01:44:41] <Krinkle>	 No, I don't think we have any reason to believe a query problem happened
[01:52:55] <AndyRussG>	 mmhmm
[01:53:05] <AndyRussG>	 Gotta afk, back in about 15m!
[01:53:17] <ejegg>	 should I dump all the other module versions too?
[02:30:11] <RoanKattouw>	 ejegg: Can't hurt too much I suppose
[02:30:28] <RoanKattouw>	 ejegg: But maybe dump choiceData separately
[02:42:47] <AndyRussG|brb>	 aaaarg much longer to return here, back soon I hope
[02:53:38] <ejegg>	 yep, I'm doing a separate curl request for the startup module and parsing the versions out of that
[02:56:44] <RoanKattouw>	 OK cool
[02:56:59] <RoanKattouw>	 Are you also doing a separate curl for the contents of the choiceData module?
[02:57:49] <RoanKattouw>	 OK I got phantomjs to work on my AWS instance BTW
[02:57:54] <RoanKattouw>	 In case you hadn't gotten it working already
[02:59:33] <ejegg>	 RoanKattouw: yeah, also doing that module request separately
[02:59:43] <RoanKattouw>	 Cool
[02:59:46] <ejegg>	 in case anything makes phantomjs die before I can parse it
[03:00:14] <RoanKattouw>	 I'm running those same things on my AWS instance, every 5 mins starting at :59 just now
[03:00:17] <ejegg>	 or rather, before it defines that stuff in mediaWiki.centralNotice
[03:00:23] <RoanKattouw>	 For redundancy in case your thing dies or whatever
[03:00:27] <ejegg>	 thanks!
[03:01:17] * AndyRUssG bets on no outages
[03:01:22] <AndyRUssG>	 A watch outage never boils?
[03:01:39] <ejegg>	 got it running on 2 machines here, one on the 0s & 5s (vpned thru SF) and one on the 3s & 8s that geolocates to New Orleans
[03:01:45] <RoanKattouw>	 Nice
[03:01:50] <RoanKattouw>	 I have one in us-east-1 (Virginia)
[03:01:59] <RoanKattouw>	 So probably like across the street from eqiad or something
[03:02:07] <AndyRussG>	 Hmmm
[03:02:08] <ejegg>	 cool, so we should at least hit eqiad and ulsfo
[03:02:26] <RoanKattouw>	 Maybe codfw too with your NOLA one
[03:02:36] <RoanKattouw>	 Mine isn't grabbing the entire startup module though
[03:02:50] <AndyRussG>	 ejegg: RoanKattouw: maybe also run some of these in languages that weren't affected?
[03:03:01] <RoanKattouw>	 Hmm good idea
[03:03:03] <ejegg>	 AndyRussG: ah, could do
[03:03:05] <RoanKattouw>	 What's the set of langs we should run?
[03:03:07] <ejegg>	 es was unaffected
[03:03:10] <AndyRussG>	 (but that are getting campaigns)
[03:03:23] <ejegg>	 and is targeted by a campaign
[03:03:28] <ejegg>	 also cs
[03:03:34] <AndyRussG>	 mmm
[03:04:19] <ejegg>	 hehe, that Fla-Vor-Ice article is getting an unearned popularity spike
[03:04:39] <ejegg>	 hope analytics filters on the phantomjs useragent
[03:05:06] <RoanKattouw>	 And curl
[03:05:13] <RoanKattouw>	 Hmm I guess I'm only using curl for load.php so that's OK
[03:07:04] <RoanKattouw>	 OK, so en and es, maybe cs
[03:07:08] <RoanKattouw>	 Any other ones that /were/ affected?
[03:07:19] <RoanKattouw>	 Maybe we should do 2 affected and 2 unaffected ones?
[03:08:05] <ejegg>	 affected: de, en, fr, it, simple
[03:08:13] <RoanKattouw>	 OK
[03:08:19] <RoanKattouw>	 So I'll hit en, de, es, cs
[03:09:04] <AndyRussG>	 Probably it makes sense to try digging into the data some more
[03:09:17] <ejegg>	 hmm, that script should take a language arg
[03:09:25] <ejegg>	 and hit Special:Random
[03:09:40] <AndyRussG>	 Comparing yesterday's and today's outages might yield some info
[03:10:14] <AndyRussG>	 If it were just some random infrastructure problem giving out corrupt data, why would it have hit the same languages both days?
[03:11:07] <AndyRussG>	 Unless that's just coincidence, or just because of what we're targeting somehow...
[03:11:44] <ejegg>	 thanks for all the help RoanKattouw !
[03:11:54] <ejegg>	 I'm going to step away from the screens for a while
[03:15:10] <RoanKattouw>	 Alright I've got it all running now
[03:15:44] <RoanKattouw>	 Capturing your phantom script and also the choiceData module and the startup module for en, es, de and cs at the 4s and 9s from Virginia
[03:16:36] <RoanKattouw>	 I'm gonna head home now
[03:16:56] <RoanKattouw>	 We'll see what comes out of this tomorrow
[03:48:56] <AndyRussG>	 RoanKattouw: ejegg: what about a VPS that we force to use IPv6?
[03:52:44] <RoanKattouw>	 AndyRussG: ooh great idea
[03:53:05] <RoanKattouw>	 When I get home I'll check if my EC2 instance uses v6
[04:02:05] <grrrit-wm>	 (PS1) Cdentinger: Restrict financial_txn rows [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325085 (https://phabricator.wikimedia.org/T152238)
[04:52:54] <AndyRussG>	 RoanKattouw: I think I have that option at DigitalOcean
[04:56:53] <RoanKattouw>	 AndyRussG: I'll check if mine is IPv6 but I can also give you the command I'm running
[04:58:02] <RoanKattouw>	 while true; do for lang in en es de cs; do curl https://$lang.wikipedia.org/w/load.php?modules=ext.centralNotice.choiceData -o choiceData-$lang-$(date +%H%M);  QT_QPA_PLATFORM=offscreen phantomjs ../phantom-test.js $lang > phantom-$lang-$(date +%H%M); done; sleep 300; done;
[04:58:11] <RoanKattouw>	 Where ../phantom-test.js is https://phabricator.wikimedia.org/paste/edit/4559/
[04:59:56] <RoanKattouw>	 Hmph no I'm using v4
[05:00:16] <RoanKattouw>	 I can run curl with -6 though
[05:00:42] <RoanKattouw>	 AndyRussG: But maybe you can just run the same command as me on your IPv6-only VPS
[05:01:03] <RoanKattouw>	 An easy way to check which one is being used is curl 'https://en.wikipedia.org/w/api.php?action=query&meta=userinfo&format=json'
[05:01:27] <RoanKattouw>	 Which, ironically I have IPv6 at my house and not on my VPS
[05:17:13] <Krinkle>	 RoanKattouw: Nice script and neat. I guess the way you beat the async problem is by re-checking whenever a resource is received.
[05:17:22] <RoanKattouw>	 Hm?
[05:17:37] <RoanKattouw>	 Oh you mean the timestamping?
[05:17:44] <Krinkle>	 RoanKattouw: the phantom script
[05:17:48] <RoanKattouw>	 Not mine
[05:17:57] <RoanKattouw>	 That's all ejegg|away
[05:18:16] <Krinkle>	 ah, okay
[05:18:32] <RoanKattouw>	 Oh I forgot to add my modification that substitutes args[1] for 'en'
[05:18:55] <RoanKattouw>	 AndyRussG: --^^ you will need a small modification to phantom-test.js if you are indeed going to run it
[05:21:08] <Krinkle>	 I would've probably done something like evaluate() -> RLQ.push -> mw.loader.using -> callPhantom(data)
[05:21:30] <Krinkle>	 which can time out or be idle in case of errors
[06:04:06] <wikibugs_>	 Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice, Performance-Team, Spike: Spike: CentralNotice: Investigate new ways to implement cross-domain banner close-button functionality, to end the Special:HideBanners cookie storm - https://phabricator.wikimedia.org/T117433#1774073 (...
[06:38:28] <wikibugs_>	 Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice, Performance-Team, Spike: Spike: CentralNotice: Investigate new ways to implement cross-domain banner close-button functionality, to end the Special:HideBanners cookie storm - https://phabricator.wikimedia.org/T117433#1774073 (...
[06:53:49] <wikibugs_>	 Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice, Performance-Team, Spike: Spike: Investigate alternatives to Special:HideBanners cookie storm for cross-domain banner close-button - https://phabricator.wikimedia.org/T117433#2843761 (Krinkle)
[06:54:06] <Krinkle>	 (now the message doesn't get cut off)
[07:19:20] <AndyRussG>	 Krinkle: know how to force calls from a VPS to WMF prod to go thru ipv6 for this script?
[08:17:36] <AndyRussG>	 No dropoff so far....
[08:18:02] <AndyRussG>	 https://grafana.wikimedia.org/dashboard/db/kafka-by-topic?from=1480507894109&to=1480753064257&var-cluster=analytics-eqiad&var-kafka_brokers=All&var-topic=eventlogging_CentralNoticeBannerHistory
[08:52:53] <ejegg|away>	 watched pot, huh?
[14:25:53] <AndyRussG>	 nope no outage! /me celebrates?
[15:59:38] <wikibugs_>	 Fundraising Sprint Waiting for Godot, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Unplanned-Sprint-Work: Central Notice: possible CN issue early on December 1st UTC - https://phabricator.wikimedia.org/T152122#2844144 (Ejegg) A watched pot never boils. No banner dip observed December...
[16:13:32] <grrrit-wm>	 (CR) Ejegg: [C: 2] Restrict financial_txn rows [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325085 (https://phabricator.wikimedia.org/T152238) (owner: Cdentinger)
[16:17:48] <grrrit-wm>	 (Merged) jenkins-bot: Restrict financial_txn rows [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325085 (https://phabricator.wikimedia.org/T152238) (owner: Cdentinger)
[18:00:02] <slander>	 fr-tech: The Great Movie Posters:
[18:00:03] <slander>	 A mis-spawned murderous abomination from the nether reaches of an
[18:00:03] <slander>	 unimaginable hell.
[18:00:03] <slander>	                 -- The Killer of Castle Brood (1967)
[18:00:03] <slander>	 NEW -- SICKENING HORROR to make your STOMACH TURN and FLESH CRAWL!
[18:00:03] <slander>	                 -- Frankenstein's Bloody Terror (1968)
[18:00:03] <slander>	 LUST-MAD MEN AND LAWLESS WOMEN IN A VICIOUS AND SENSUOUS ORGY OF
[18:00:04] <slander>	 SLAUGHTER!
[18:00:04] <slander>	                 -- Five Bloody Graves (1969)
[18:00:05] <slander>	 The family that slays together stays together.
[18:00:05] <slander>	                 -- Bloody Mama (1970)
[18:00:06] <slander>	  -- discuss.
[23:28:30] <wikibugs>	 Fundraising-Analysis, Fundraising-Backlog: Make Github wmf-fr repository public - https://phabricator.wikimedia.org/T151983#2834119 (Peachey88) It should probably be transitioned over to our Gerrit or Diffusion repos, I see no reason for this to be on GitHub in the first place…