[00:39:27] <shinken-wm>	 PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[01:19:23] <shinken-wm>	 RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0]
[01:45:24] <shinken-wm>	 PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[03:33:04] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Growth-Team, 10MediaWiki-extensions-PageCuration, 10Collaboration-Team-Triage (Collab-Team-Next-Quarter): [betalabs]: Page triage: "Uncaught TypeError: Cannot read property 'getLogPageTitle' of undefined" for 'Redirects for discussi... - https://phabricator.wikimedia.org/T196954
[06:27:44] <kart_>	 !log Updated cxserver to f8c71a1
[06:27:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:03:38] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Release Pipeline, 10Jenkins: Enable MFA on Jenkins - https://phabricator.wikimedia.org/T198814 (10hashar)
[08:06:30] <wikibugs>	 (03PS1) 10Hashar: Migrate Graph to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/443908 (https://phabricator.wikimedia.org/T183512)
[08:06:55] <wikibugs>	 (03CR) 10Hashar: [C: 032] Migrate Graph to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/443908 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar)
[08:08:22] <wikibugs>	 (03Merged) 10jenkins-bot: Migrate Graph to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/443908 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar)
[08:11:50] <shinken-wm>	 PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[08:42:58] <wikibugs>	 (03PS1) 10Hashar: Migrate PageViewInfo to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/443915 (https://phabricator.wikimedia.org/T183512)
[08:43:21] <wikibugs>	 (03CR) 10Hashar: [C: 032] Migrate PageViewInfo to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/443915 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar)
[08:44:53] <wikibugs>	 (03Merged) 10jenkins-bot: Migrate PageViewInfo to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/443915 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar)
[08:46:48] <shinken-wm>	 RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[08:48:32] <wikibugs>	 (03PS1) 10Hashar: Update Quibble QA report, gate is migrated [integration/config] - 10https://gerrit.wikimedia.org/r/443916 (https://phabricator.wikimedia.org/T197469)
[08:48:54] <wikibugs>	 (03CR) 10Hashar: [C: 032] Update Quibble QA report, gate is migrated [integration/config] - 10https://gerrit.wikimedia.org/r/443916 (https://phabricator.wikimedia.org/T197469) (owner: 10Hashar)
[08:50:21] <wikibugs>	 (03Merged) 10jenkins-bot: Update Quibble QA report, gate is migrated [integration/config] - 10https://gerrit.wikimedia.org/r/443916 (https://phabricator.wikimedia.org/T197469) (owner: 10Hashar)
[09:35:58] <wikibugs>	 10Phabricator, 10Security: Trusted contributors cannot escalate tasks as security issues - https://phabricator.wikimedia.org/T198831 (10MarcoAurelio) Not sure if D1075 could fix this though.
[09:39:12] <hashar>	 dockerfiles/npm-test/run.sh  :  npm run-script "{@:-test}"
[09:42:39] <Hauskatze>	 The url to the chat log in the topic on this channel gives 404 to me.
[09:46:09] <p858snake>	 Hauskatze: https://wm-bot.wmflabs.org/browser/index.php?display=%23wikimedia-releng
[09:47:45] <Hauskatze>	 thanks p858snake - this confirms a bug in wikibugs not reporting all the comments made on tasks
[09:48:23] <Hauskatze>	 I made two comments on T198831 but only the last one was written in this channel
[09:48:24] <stashbot>	 T198831: Trusted contributors cannot escalate tasks as security issues - https://phabricator.wikimedia.org/T198831
[09:49:06] <p858snake>	 check with legoktm, but i suspect that might be some anti-flood type of thing
[09:49:26] <Hauskatze>	 afair there was wikibugs  and wikibugs_ right?
[09:52:19] <wikibugs>	 (03PS1) 10Hashar: (WIP) job to run wdio tests for a single repo [integration/config] - 10https://gerrit.wikimedia.org/r/443931
[09:55:13] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] (WIP) job to run wdio tests for a single repo [integration/config] - 10https://gerrit.wikimedia.org/r/443931 (owner: 10Hashar)
[09:56:58] <wikibugs>	 (03PS2) 10Hashar: (WIP) job to run wdio tests for a single repo [integration/config] - 10https://gerrit.wikimedia.org/r/443931
[10:15:42] <wikibugs>	 10Phabricator, 10Security: Trusted contributors cannot escalate tasks as security issues - https://phabricator.wikimedia.org/T198831 (10MarcoAurelio)
[10:42:55] <wikibugs>	 10Scap, 10WorkType-NewFunctionality: Add a --labs option to 'scap update-interwiki-cache' to be able to update the interwiki-labs.php file using Scap - https://phabricator.wikimedia.org/T198844 (10MarcoAurelio)
[11:07:20] <wikibugs>	 (03PS3) 10Zfilipin: (WIP) job to run wdio tests for a single repo [integration/config] - 10https://gerrit.wikimedia.org/r/443931 (https://phabricator.wikimedia.org/T188742) (owner: 10Hashar)
[11:17:55] <wmf-insecte>	 Yippee, build fixed!
[11:17:56] <wmf-insecte>	 Project beta-scap-eqiad build #214593: 09FIXED in 4 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214593/
[11:18:52] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie for all repositores with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilipin)
[11:19:23] <wikibugs>	 10Release-Engineering-Team, 10GitHub-Mirrors, 10Security-Team: Enforce 2FA for GitHub members - https://phabricator.wikimedia.org/T198810 (10Reedy)
[11:26:00] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie for all repositores with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilipin)
[12:18:06] <wikibugs>	 10Diffusion, 10Phabricator: Turn off 'blame' by default on Diffusion - https://phabricator.wikimedia.org/T198838 (10Aklapper) According to https://secure.phabricator.com/rPafc3099ee785c0f9cb86c702fc1d04d2be9f6af5 there is  a "Hide Blame" option in the "View Options" dropdown so it can be toggled off?
[12:42:25] <wikibugs>	 10Phabricator, 10WMSE-Bug-Reporting-and-Translation-2018: "Create Task" in favorites menu should be called "Create Task (Simple)" - https://phabricator.wikimedia.org/T198837 (10Lokal_Profil) >>! In T198837#4399050, @Aklapper wrote: > Thanks. Done.  Wow that was quick!. Thanks
[14:28:15] <wikibugs>	 (03PS1) 10Hashar: Archive ContributorsAddon extension [integration/config] - 10https://gerrit.wikimedia.org/r/443976 (https://phabricator.wikimedia.org/T198864)
[14:28:32] <wikibugs>	 (03CR) 10Hashar: [C: 032] Archive ContributorsAddon extension [integration/config] - 10https://gerrit.wikimedia.org/r/443976 (https://phabricator.wikimedia.org/T198864) (owner: 10Hashar)
[14:30:16] <wikibugs>	 (03Merged) 10jenkins-bot: Archive ContributorsAddon extension [integration/config] - 10https://gerrit.wikimedia.org/r/443976 (https://phabricator.wikimedia.org/T198864) (owner: 10Hashar)
[14:30:18] <wikibugs>	 (03PS1) 10Hashar: Migrate BreadCrumbs extension to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/443977 (https://phabricator.wikimedia.org/T183512)
[14:30:27] <wikibugs>	 (03CR) 10Hashar: [C: 032] Migrate BreadCrumbs extension to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/443977 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar)
[14:32:06] <wikibugs>	 (03Merged) 10jenkins-bot: Migrate BreadCrumbs extension to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/443977 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar)
[15:18:59] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace._srv.byte_percentfree (<40.00%)
[15:45:57] <wikibugs>	 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar)
[16:00:14] <wikibugs>	 (03PS4) 10Zfilipin: (WIP) job to run wdio tests for a single repo [integration/config] - 10https://gerrit.wikimedia.org/r/443931 (https://phabricator.wikimedia.org/T188742) (owner: 10Hashar)
[16:03:57] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-slave-jessie-1002 is OK: OK: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)
[16:09:38] <shinken-wm>	 PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:19:29] <wikibugs>	 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar)
[16:20:04] <wikibugs>	 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar)
[17:00:30] <wikibugs>	 (03PS1) 10Hashar: jjb: add support to pass arguments to quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444024 (https://phabricator.wikimedia.org/T196960)
[17:11:02] <wikibugs>	 (03PS1) 10Hashar: Quibble jobs that skip selenium entirely [integration/config] - 10https://gerrit.wikimedia.org/r/444028 (https://phabricator.wikimedia.org/T196960)
[17:21:10] <paladox>	 hmm i get "Context not available."
[17:21:14] <paladox>	 on https://phabricator.wikimedia.org/D1075
[17:21:18] <paladox>	 twentyafterfour ^^
[17:22:36] <twentyafterfour>	 paladox: probably a diff uploaded via web ui rather than arc
[17:22:41] <paladox>	 oh
[17:22:45] <paladox>	 ok
[17:28:29] <Hauskatze>	 if it is a diff of mine, yep, I upload them using the web UI
[17:28:33] <Hauskatze>	 I don't have arc
[17:33:20] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T191060 (10greg) a:03hashar
[17:33:34] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T191061 (10greg) a:03hashar
[17:49:35] <shinken-wm>	 RECOVERY - SSH on integration-slave-docker-1017 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0)
[17:53:12] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Growth-Team, 10MediaWiki-extensions-PageCuration, 10Collaboration-Team-Triage (Collab-Team-Next-Quarter): [betalabs]: Page triage: "Uncaught TypeError: Cannot read property 'getLogPageTitle' of undefined" for 'Redirects for discussi... - https://phabricator.wikimedia.org/T196954
[17:53:15] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Growth-Team (Current Sprint): Set up PageTriage test environment in beta labs - https://phabricator.wikimedia.org/T198898 (10Etonkovidova)
[18:22:18] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Wikibugs, 10Patch-For-Review: Deprecate -devtools and redirect to -releng? - https://phabricator.wikimedia.org/T185285 (10greg) >>! In T185285#4262451, @Peachey88 wrote: > I don't see a need in redirecting the channel/closing it, I've actually seen it a few times when p...
[18:27:51] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Kanban): Make sure elasticsearch 6 is supported in phabricator - https://phabricator.wikimedia.org/T181393 (10greg) p:05Lowest>03Normal Re-prioritizing.  @EBernhardson still no date yet, right? And from what you said in T192614#4249710 are we actually OK with curr...
[19:38:42] <Krenair>	 The last Puppet run was at Thu Jul  5 13:31:12 UTC 2018 (367 minutes ago). Puppet is disabled. reason not specified
[19:38:43] <Krenair>	 who did this
[19:40:25] <Krenair>	 Jul  5 13:33:12 deployment-elastic07 sudo:    gehel : TTY=pts/0 ; PWD=/home/gehel ; USER=root ; COMMAND=/usr/bin/puppet agent --disable
[19:41:19] <Krenair>	 gehel why is there no reason specified?
[19:42:47] <gehel>	 Krenair: probably because I forgot to add one ;)
[19:43:02] <gehel>	 I'll check in a minute. Thanks for the ping !
[19:43:41] <Krenair>	 thanks
[19:46:15] <gehel>	 looks like I disabled it with a reason first, but re-disabled it without the second time. Bad gehel...
[20:06:48] <jdlrobson>	 I'm getting "You ("x.x.x.241") are issuing too many requests too quickly." when I visit phab
[20:06:56] <jdlrobson>	 Is this related to the spam attack?
[20:07:00] <jdlrobson>	 I'm logged in so this seems weird to me
[20:07:22] <jdlrobson>	 It's stopping me from doing work right now 
[20:09:20] <apergos>	 are you at home or in the office?
[20:10:00] <jdlrobson>	 office wifi
[20:10:06] <jdlrobson>	 so guessing it's everyone in the office?
[20:10:11] <apergos>	 there is rate limiting, maybe the office ip....
[20:10:13] <jdlrobson>	 i cant even access incognito
[20:10:42] <apergos>	 web interface right?
[20:11:05] <jdlrobson>	 yup
[20:11:13] <jdlrobson>	 i confirmed others in the office cannot access either
[20:11:18] <apergos>	 see -staff
[20:11:24] <apergos>	 just got another report
[20:11:39] <apergos>	 well the good news is that rate limiting is now working
[20:11:42] <apergos>	 the bad news...
[20:12:07] <paladox>	 your all blocked
[20:12:08] <paladox>	 lol
[20:12:31] <paladox>	 hmm, we should implement a way to whitelist users
[20:12:41] <paladox>	 instead of ip as ips change all the time/
[20:13:13] <apergos>	 can you allow the office ip to bypass the rate limit?
[20:13:20] <paladox>	 yes
[20:13:27] <paladox>	 there's a whitelist
[20:13:34] <paladox>	 at least the WMDE was whitelisted
[20:13:36] <apergos>	 that will do for letting people get work done right now at least
[20:14:34] <paladox>	 twentyafterfour ^^
[20:15:24] <apergos>	 the office ip has been posted in the other channel so it can be added
[20:17:05] <Krenair>	 is it tan1.corp.wikimedia.org = 198.73.209.1 by any chance?
[20:17:22] <Krenair>	 that one's public knowledge
[20:19:27] <apergos>	 no
[20:32:57] <apergos>	 https://phabricator.wikimedia.org/T198612   according to this they are not exempt yet
[20:33:05] <apergos>	 I hope we have a mechanism for this
[20:37:42] <paladox>	 apergos they were whitelisted since then
[20:38:07] <apergos>	 I wish there were a note on the task to that effect
[20:38:11] <apergos>	 and especially how it was done
[20:38:57] <paladox>	 i created https://phabricator.wikimedia.org/T198614
[20:40:27] <paladox>	 wikibugs seems affected too.
[20:40:41] <paladox>	 i only see it post gerrit results
[21:16:36] <apergos>	 as it's midnight here and I have no idea how the wmde fix was done
[21:16:45] <apergos>	 I'm going to go get some sleep
[21:17:18] <Krenair>	 was there a wmde fix done?
[21:17:26] <apergos>	 scroll up
[21:17:30] <apergos>	 paladox says yes
[21:18:58] <p858snake>	 has anyone filed a task or emailed ops to look at the office one? I wouldn't rely on scrollback for that
[21:19:26] <Krenair>	 yes
[21:19:31] <Krenair>	 I emailed ops with a link to the task
[21:23:48] <Hauskatze>	 Krenair: can you please have a look at T198891 ? I see that you are a project maintainer for that bot.
[21:23:49] <stashbot>	 T198891: wikibugs malfunctioning - https://phabricator.wikimedia.org/T198891
[21:24:00] <Krenair>	 maintainer is a strong word
[21:24:17] <Hauskatze>	 s/maintainer/have access to/
[21:24:17] <wikibugs>	 (03CR) 10Arlolra: "Hashar, this is the other part of https://gerrit.wikimedia.org/r/#/c/integration/config/+/442004/" [integration/config] - 10https://gerrit.wikimedia.org/r/442002 (owner: 10Arlolra)
[21:24:23] <Krenair>	 can you give an example of something it's missed
[21:25:06] <Krenair>	 what should have happened, what did happen, how to reproduce it, etc.
[21:25:22] <Hauskatze>	 Yes. This morning I posted some comments at a task which is reported to this channel. Wikibugs should have therefore reported the activity here. It didn't but just one.
[21:25:30] <Hauskatze>	 I can fetch the task.
[21:25:43] <Krenair>	 It didn't but just one?
[21:25:57] <Hauskatze>	 Also, zhuyifei1999_ and I had a discussion on #pywikibot and he thinks wikibugs is missing some activity as well
[21:25:59] <paladox>	 wikibugs is likly hitting the rate limiting?
[21:26:12] <paladox>	 i filled a task for that too
[21:26:23] <Hauskatze>	 paladox: maybe because his wikibugs_ brother isn't here?
[21:26:30] <paladox>	 Nope
[21:26:36] <paladox>	 wikibugs_ is for flooding.
[21:26:48] <paladox>	 but it works with gerrit
[21:26:51] <paladox>	 https://phabricator.wikimedia.org/T198915
[21:27:00] <Krenair>	 yeah I don't think that has any effect on the process that pulls phab tasks into redis
[21:27:01] <Hauskatze>	 ah, because on a channel I am we have two bots to report stuff to avoid flooding off
[21:27:53] <Hauskatze>	 okay so it's T198915 apparently, right?
[21:27:53] <stashbot>	 T198915: Exclude wiki bugs from phabricator rate limit - https://phabricator.wikimedia.org/T198915
[21:27:57] <wikibugs>	 10Phabricator, 10Wikibugs: Exclude wiki bugs from phabricator rate limit - https://phabricator.wikimedia.org/T198915 (10Krenair) wikibugs runs on labs and I don't think that's going to get whitelisted.
[21:30:02] <wikibugs>	 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review, 10User-notice: Make PolyGerrit the default ui - https://phabricator.wikimedia.org/T196812 (10Johan) OK, I simplified a bit and geared it a bit more towards the main audience of Tech News (non-developers), but it's included.
[21:30:37] <wikibugs>	 10Continuous-Integration-Config, 10Gerrit, 10Jenkins: Jenkins bot ignores the patches submitted by Mpaa - https://phabricator.wikimedia.org/T198573 (10Mpaa) 05Open>03Resolved a:03Mpaa Seems OK now. Thanks and sorry for the inconvenience.
[21:34:16] <Krenair>	 Hauskatze, looking at tools-bastion-03.tools.eqiad.wmflabs:/data/project/wikibugs/wikibugs.log
[21:34:24] <twentyafterfour>	 ...
[21:34:48] <Krenair>	 there are exceptions going in there
[21:34:49] <Krenair>	 sup twentyafterfour 
[21:34:51] <twentyafterfour>	 We might as well just disable to throttle for now, it's too hard to whitelist individual ips
[21:35:29] <Krenair>	 like technically too hard?
[21:36:15] <twentyafterfour>	 well not technically very hard but the list will keep expanding of who needs to be whitelisted and there is no convenient or clean place to put it in the code, so it'll be hacks on top of hacks unless it's done right
[21:36:30] <twentyafterfour>	 doing it right takes a little more time and that doesn't alleviate the situation immediately
[21:36:39] <Hauskatze>	 I envy you guys. I wish I had more technical knowledge :(
[21:36:47] <Hauskatze>	 Respect.
[21:37:21] <Krenair>	 twentyafterfour, right now we're talking about the WMDE and WMF offices
[21:37:58] <Krenair>	 I don't think we're necessarily opening the floodgates
[21:39:03] <twentyafterfour>	 Krenair: true but it's still a mess to add exceptions - the code has no support for whitelisting so I'm not sure how to add it
[21:39:28] <Krenair>	 this is phabricator we were somewhat expecting this when implementing rate limiting weren't we?
[21:40:25] <twentyafterfour>	 Krenair: where can I see the IPs that need whitelisting?
[21:40:55] <twentyafterfour>	 Krenair: it had rate limits previously. I'm not sure what the parameters were though 
[21:41:04] <twentyafterfour>	 they must have been higher than now
[21:41:23] <Krenair>	 https://phabricator.wikimedia.org/T198612 which I copied into my potential patch for it
[21:42:02] <Krenair>	 I wouldn't be surprised if there are more IPs for those offices
[21:42:08] <twentyafterfour>	 yeah \
[21:42:29] <Krenair>	 but some people try to shroud the corp network in secrecy so
[21:42:39] <Reedy>	 There used to be a few different
[21:42:41] * Krenair shrugs
[21:42:45] <Reedy>	 I dunno if/what changed when the office moved
[21:42:48] <Krenair>	 tan[1-4] ?
[21:42:56] <Krenair>	 and guest-tan1
[21:43:07] <Krenair>	 now there's tan241
[21:43:42] <twentyafterfour>	 I think Krenair's patch looks like a good solution for now
[21:44:03] <twentyafterfour>	 or disable throttle
[21:44:40] <Krenair>	 Hauskatze, so wikibugs
[21:44:55] <Krenair>	 here's a common exception
[21:45:18] <Hauskatze>	 I'm listening.
[21:45:34] <twentyafterfour>	 I tried to figure out how to exempt wikibugs from the throttle and I couldn't find a clean solution because phabricator doesn't always have a logged in session when it processes the throttle scoring
[21:45:44] <Krenair>	 https://phabricator.wikimedia.org/P7331
[21:46:00] <Krenair>	 I don't think we actually know that wikibugs is running into a rate limit yet twentyafterfour 
[21:46:04] <twentyafterfour>	 we could reduce the rate of polling maybe?
[21:46:18] <Krenair>	 although it's certainly possible given its nature
[21:46:40] <twentyafterfour>	 It polls pretty quickly so I'd think that it is hitting the throttle
[21:47:19] <twentyafterfour>	 We could either raise the max score or slow down the poll loop in wikibugs
[21:48:10] <twentyafterfour>	 doubling the current max score would probably be a good start
[21:50:41] <Krenair>	   File "/mnt/nfs/labstore-secondary-tools-project/wikibugs/py-wikibugs2/lib/python3.4/site-packages/phabricator/__init__.py", line 49, in connect
[21:50:41] <Krenair>	     assert req.status_code == 200 # krenair live hack
[21:50:41] <Krenair>	 AssertionError
[21:52:09] <Krenair>	 alright so let's see what the body of the response is when the code isn't 200
[21:52:15] <Krenair>	 there are some other types coming in too btw
[21:52:21] <Krenair>	 yeah here we go
[21:52:27] <Krenair>	   File "/mnt/nfs/labstore-secondary-tools-project/wikibugs/py-wikibugs2/lib/python3.4/site-packages/phabricator/__init__.py", line 50, in connect
[21:52:27] <Krenair>	     raise Exception(req.content)
[21:52:27] <Krenair>	 Exception: b'TOO MANY REQUESTS\nYou ("10.68.18.17") are issuing too many requests too quickly.\n'
[21:55:22] <Krenair>	 twentyafterfour, made it sleep for 2 seconds instead of 1 second between polls
[21:56:58] <twentyafterfour>	 the limit is 5 per minute
[21:57:09] <twentyafterfour>	 so that needs to be more than 2 seconds really
[21:58:02] <Krenair>	 ah
[21:58:18] <twentyafterfour>	 5 per minute for 5 consecutive minutes
[21:58:41] <Krenair>	 okay so 12 seconds
[21:59:12] <twentyafterfour>	 that seems reasonable. I don't think wikibugs needs to respond within 2 seconds to every change in phab
[22:00:08] <Krenair>	 still getting different types of exceptions. let's see
[22:00:38] <Krenair>	 oh wow
[22:00:58] <Krenair>	         data_dict_str = task_page.split(
[22:00:59] <Krenair>	             '<script type="text/javascript">JX.Stratcom.mergeData(0,'
[22:00:59] <Krenair>	         )[1].split(
[22:00:59] <Krenair>	             ");\nJX.onload"
[22:00:59] <Krenair>	         )[0]
[22:01:08] <Krenair>	 cannot unsee
[22:02:40] <wikibugs>	 10Phabricator: Attempting to create a new Conpherence room produces an internal server error - https://phabricator.wikimedia.org/T198923 (10Dinoguy1000)
[22:03:34] <wikibugs>	 10Phabricator: Attempting to create a new Conpherence room produces an internal server error - https://phabricator.wikimedia.org/T198923 (10Krenair) I doubt it was you I think you just had unfortunate timing: ```18<bd80818> twentyafterfour: 500 errors for me from Phab following that !log 18<bd80818>...
[22:13:10] <wikibugs>	 10Phabricator: Attempting to create a new Conpherence room produces an internal server error - https://phabricator.wikimedia.org/T198923 (10Dinoguy1000) I'd be really happy with it being unfortunate timing. I guess I'll retry creating the room and see what happens, then.
[22:14:56] <wikibugs>	 10Phabricator: Attempting to create a new Conpherence room produces an internal server error - https://phabricator.wikimedia.org/T198923 (10Dinoguy1000) 05Open>03Invalid Room created successfully; closing as invalid. Thanks for the explanation, @Krenair! =)
[22:41:35] <wikibugs>	 10Phabricator, 10Trusted-Contributors: Whitelist the Trusted-Contributors group from rate limit - https://phabricator.wikimedia.org/T198614 (10mmodell) This isn't technically feasible. The rate limit is based on IP and the throttle doesn't necessarily know which user is associated with an IP.
[22:50:26] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Kanban): Make sure elasticsearch 6 is supported in phabricator - https://phabricator.wikimedia.org/T181393 (10mmodell) a:03mmodell
[22:53:13] <wikibugs>	 10Diffusion, 10Phabricator: Turn off 'blame' by default on Diffusion - https://phabricator.wikimedia.org/T198838 (10mmodell) I'm not sure why this is happening.
[23:05:05] <wikibugs>	 10Phabricator: Replace project names by PHIDs in our Phab extensions - https://phabricator.wikimedia.org/T198589 (10mmodell) The problem with using PHIDs is that it makes testing difficult - on my test instance the PHIDs differ from production. I suppose I could fix it manually in the database though.