[03:58:35] Yippee, build fixed! [03:58:35] Project selenium-MultimediaViewer » firefox,mediawiki,Linux,BrowserTests build #327: 09FIXED in 2 min 34 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=mediawiki,PLATFORM=Linux,label=BrowserTests/327/ [07:03:11] PROBLEM - Puppet run on deployment-imagescaler01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:49:08] PROBLEM - Puppet run on buildlog is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:44:46] Yippee, build fixed! [15:44:47] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #356: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/356/ [17:31:54] 10Browser-Tests-Infrastructure, 07Easy, 05MW-1.29-release (WMF-deploy-2017-03-14_(1.29.0-wmf.16)), 13Patch-For-Review: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#3093950 (10Harjotsingh) @zeljkofilipin Removed the lines from all files except the .gitignore files [20:41:46] Yippee, build fixed! [20:41:46] Project selenium-Echo » chrome,beta,Linux,BrowserTests build #331: 09FIXED in 45 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/331/ [20:41:54] Yippee, build fixed! [20:41:55] Project selenium-Echo » firefox,beta,Linux,BrowserTests build #331: 09FIXED in 54 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/331/ [20:58:58] 06Release-Engineering-Team, 10Phabricator: Logged out users see tasks locked - https://phabricator.wikimedia.org/T160302#3094362 (10Luke081515) [21:31:01] 06Release-Engineering-Team, 10Phabricator: Logged out users see tasks locked - https://phabricator.wikimedia.org/T160302#3094408 (10Paladox) [21:40:19] 10Gerrit, 06Release-Engineering-Team, 10Mail, 06Operations: Gerrit emails are showing up as being sent late - https://phabricator.wikimedia.org/T159960#3084643 (10valhallasw) From P5028 : ``` Received: from 127.0.0.1 (EHLO mx1001.wikimedia.org) (208.80.154.76) by mta1470.mail.gq1.yahoo.com with SMTPS;... [22:02:19] 10Gerrit, 06Release-Engineering-Team, 10Mail, 06Operations: Gerrit emails are showing up as being sent late - https://phabricator.wikimedia.org/T159960#3094454 (10Paladox) Filled here https://forums.yahoo.net/t5/Sending-and-receiving/Receiving-emails-from-some-of-wikipedia-s-domains-is-taking/m-p/203788/hi... [22:04:03] 10Gerrit, 06Release-Engineering-Team, 10Mail, 06Operations: Gerrit emails are showing up as being sent late - https://phabricator.wikimedia.org/T159960#3094456 (10Paladox) Other users have reported it here https://forums.yahoo.net/t5/Sending-and-receiving/Delays-in-receiving-emails/m-p/205982/highlight/fal... [22:19:55] paladox: I can try sending you a test email if you want [22:20:19] although, I'd dump yahoo mail [22:20:23] too much broken, sadly [22:33:28] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:53:22] Platonides hi, sending an email using the mail command on the mac works [22:53:36] Seems like rate limiting per https://phabricator.wikimedia.org/T159960 [23:23:14] I wondered if they were grayslisting [23:52:49] Platonides maybe though they will be lossing more users if they keep up the throttling.