[00:22:01] 10Release-Engineering-Team, 10JobRunner-Service, 10Performance-Team, 10Regression: Investigate 30x increase in Jobrunner errors - https://phabricator.wikimedia.org/T171371#3511004 (10aaron) Did you check redis? Are their jobs back in the queues? It's possible some remote wiki is inserting jobs via JobSpeci... [00:29:19] Is there a reason that pageforms is not installed on mediawiki.org? I vaguely recall that pageforms is not well loved but I don't know why that is. I can't think of a better way to resolve https://phabricator.wikimedia.org/T171940 than using a form to edit the templates on the deployment schedule page. The only other option I can come up with is moving swat scheduling off-wiki and into phabricator [00:34:13] twentyafterfour: yeah mean wikitech? [00:34:35] greg-g: doh! yeah I guess I do [00:34:37] heh [00:35:04] we should really migrate wikitech over to mediawiki.org, it hardly deserves to be separate [00:36:32] no comment on that, but yes, pageforms and other semantic mediawiki extensions are actively discouraged against deploying in production. There is active work to remove all remnant from wikitech (via moving to striker) [01:11:29] so I should just focus on a custom phabricator form then? [01:11:38] that is, when I even have time to work on it [01:11:44] it's obviously not high priority [01:12:42] well, there's: https://phabricator.wikimedia.org/T171940 [01:12:57] ah, you commented [01:13:12] I wonder what marco has in mind re design/arch [03:56:34] 10Deployment-Systems, 10wikitech.wikimedia.org, 10User-MarcoAurelio: Create a Gadget to easily add/remove/modify patches for SWAT at wikitech:Deployments - https://phabricator.wikimedia.org/T171940#3480850 (10Legoktm) If it were possible to use VE to edit the Deployments page I think all of the pain would in... [04:21:49] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #480: 04FAILURE in 25 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/480/ [04:56:19] 10Deployment-Systems, 10wikitech.wikimedia.org, 10User-MarcoAurelio: Create a Gadget to easily add/remove/modify patches for SWAT at wikitech:Deployments - https://phabricator.wikimedia.org/T171940#3511162 (10mmodell) @legoktm: indeed, that would be a fairly good solution. The current deployment calendar inv... [05:25:10] 10Scap, 10Phabricator: scap should provide a way to skip symlink-swapping in promote - https://phabricator.wikimedia.org/T172486#3499941 (10mmodell) [05:25:22] 10Scap, 10Phabricator: scap should provide a way to skip symlink-swapping in promote - https://phabricator.wikimedia.org/T172486#3499941 (10mmodell) p:05Triage>03Normal [05:46:48] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:30:05] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [06:46:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:55:01] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:59:09] (03CR) 10Hashar: [C: 031] Fix uprightdiff debian-glue running on master [integration/config] - 10https://gerrit.wikimedia.org/r/370695 (owner: 10Legoktm) [09:06:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [09:13:19] 10Release-Engineering-Team, 10Phabricator, 10Patch-For-Review: Improve phabricator's deployment process - https://phabricator.wikimedia.org/T172847#3511439 (10Aklapper) [11:21:13] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:35:24] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Next), 10MinervaNeue, 10Readers-Web-Backlog, and 5 others: [4 hrs] MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3512073 (10bmansurov... [13:36:02] Hmm, so betacluster runs MediaWiki master but apparently not extensions master, right? [13:46:14] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #486: 04FAILURE in 2 min 13 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/486/ [13:50:08] forget that, it is working [13:50:18] Amir1: the code is live now on the beta cluster :) [13:50:44] Yess [13:50:46] Awesome [13:54:07] 10Beta-Cluster-Infrastructure, 10Multimedia, 10Thumbor: On beta commons, thumbnailing of 3D files is broken still - https://phabricator.wikimedia.org/T170444#3512155 (10Gilles) Yes, now that Thumbor is serving all thumbnail traffic, T161719 is the blocker for this task. @MarkTraceur and I got a solution work... [13:54:13] 10Beta-Cluster-Infrastructure, 10WikimediaMessages, 10Patch-For-Review: Special:ListGroupRights shows "extendedmover" literally on enwiki beta labs - https://phabricator.wikimedia.org/T169663#3512157 (10MarcoAurelio) 05Open>03Resolved a:03GeoffreyT2000 You can see it now in the beta clusted and it'll a... [13:56:18] scap = scattering crap across production <-- that made my day [13:57:19] :)))) [14:01:14] Project selenium-MinervaNeue-BugFix » firefox,beta,Linux,BrowserTests build #19: 15ABORTED in 4.1 sec: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-BugFix/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/19/ [14:01:14] Project selenium-MinervaNeue-BugFix » chrome,beta,Linux,BrowserTests build #19: 15ABORTED in 4.1 sec: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-BugFix/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/19/ [14:02:07] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Next), 10MinervaNeue, 10Readers-Web-Backlog, and 5 others: [4 hrs] MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3512208 (10Jdlrobson... [14:06:54] TabbyCat: yeah, it should be master of everything on Beta Cluster :) (within 10 minutes-ish) [14:07:36] greg-g: yep, but I didn't knew the 10 minute waiting period... I was already looking for how to scap :P [14:07:45] heh [14:07:55] thankfully not needed [14:15:16] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-CentralAuth, 10MinervaNeue, 10VisualEditor, and 2 others: Edits sometimes fail for new users erroneously reporting them as anon - https://phabricator.wikimedia.org/T172835#3512231 (10Jdlrobson) The patch is a workaround, not a fix. [14:21:03] 10MediaWiki-Releasing, 10MediaWiki-Containers, 10Wikimania-Hackathon-2017, 10Services (doing), and 2 others: Ready-to-use Docker package for MediaWiki - https://phabricator.wikimedia.org/T92826#3512282 (10dbarratt) [14:21:22] Project selenium-MinervaNeue-BugFix » chrome,beta,Linux,BrowserTests build #20: 09SUCCESS in 20 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-BugFix/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/20/ [14:26:07] %#)(%@($_@!**%#@ addshore and legoktm are killing CI right before the hackathon starts... [14:27:49] doing wikidata build changes [14:28:00] by kill I mean cause a huge backlog [14:28:37] those client and repo tests could be merged into one or sqlite and mysql run in one test :). [14:29:34] la la la la la [14:29:35] Project selenium-MinervaNeue-BugFix » firefox,beta,Linux,BrowserTests build #20: 09SUCCESS in 28 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-BugFix/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/20/ [14:49:04] 10Release-Engineering-Team (Next), 10Scap, 10Phabricator, 10Patch-For-Review: Improve phabricator's deployment process - https://phabricator.wikimedia.org/T172847#3512433 (10greg) (moving to #releng-next, feel free to move to -kanban if this is something on the shorter term) [14:54:59] 10MediaWiki-Releasing, 10MediaWiki-Containers, 10Wikimania-Hackathon-2017, 10Services (doing), and 2 others: Ready-to-use Docker package for MediaWiki - https://phabricator.wikimedia.org/T92826#3512471 (10dbarratt) a:03dbarratt [15:26:59] 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10Flow: Flow causing mediawiki-extensions-hhvm-jessie to fail - blocks merges in MobileFrontend - https://phabricator.wikimedia.org/T172811#3512650 (10thcipriani) 05Open>03Resolved a:03thcipriani >>! In T172811#3510432, @Umherirrender wr... [15:28:25] 10MediaWiki-Releasing, 10MediaWiki-Containers, 10Wikimania-Hackathon-2017, 10Services (doing), and 2 others: Ready-to-use Docker package for MediaWiki - https://phabricator.wikimedia.org/T92826#3512658 (10dbarratt) Just an update on this, deciding on where (org) the repo should be hosted and if Docker sho... [15:29:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Jenkins: Assert no PHP errors (notices, warnings) were raised or exceptions were thrown - https://phabricator.wikimedia.org/T50002#3512666 (10greg) dang (re ^ ) [15:30:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Jenkins: Assert no PHP errors (notices, warnings) were raised or exceptions were thrown - https://phabricator.wikimedia.org/T50002#516358 (10thcipriani) This assertion is now live for: mediawiki-core-qunit-sele... [15:45:31] (03PS2) 10Niedzielski: Marvin: add NPM template [integration/config] - 10https://gerrit.wikimedia.org/r/370678 (https://phabricator.wikimedia.org/T172904) [15:47:55] 10Release-Engineering-Team (Watching / External), 10JobRunner-Service, 10Performance-Team, 10Regression: Investigate 30x increase in Jobrunner errors - https://phabricator.wikimedia.org/T171371#3512779 (10greg) [16:11:28] Project selenium-MinervaNeue-BugFix » chrome,beta,Linux,BrowserTests build #22: 04FAILURE in 20 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-BugFix/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/22/ [16:17:19] (03PS1) 10Legoktm: Revert ".gitreview: swap defaultbranch for track" [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/370838 [16:17:25] (03CR) 10Legoktm: [V: 032 C: 032] Revert ".gitreview: swap defaultbranch for track" [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/370838 (owner: 10Legoktm) [16:18:10] (03PS1) 10Legoktm: Merge tag '1.1.0' into debian [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/370839 [16:18:24] (03PS1) 10Legoktm: Update for real debianizing [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/370840 [16:19:43] Project selenium-MinervaNeue-BugFix » firefox,beta,Linux,BrowserTests build #22: 04FAILURE in 29 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-BugFix/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/22/ [16:21:27] (03CR) 10jerkins-bot: [V: 04-1] Merge tag '1.1.0' into debian [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/370839 (owner: 10Legoktm) [16:24:34] (03CR) 10Legoktm: [V: 032 C: 032] "Build passes in next commit" [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/370839 (owner: 10Legoktm) [16:24:56] (03CR) 10Legoktm: [C: 032] Update for real debianizing [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/370840 (owner: 10Legoktm) [16:53:20] greg-g: anybody is doing anything with https://phabricator.wikimedia.org/T172560 "Loss of session data" on Beta Cluster [16:53:32] greg-g: the issue came back with vengeance [16:53:32] ; [16:54:15] Yippee, build fixed! [16:54:16] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #61: 09FIXED in 24 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/61/ [17:02:23] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #61: 04FAILURE in 32 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/61/ [17:18:21] 10Deployment-Systems, 10Release-Engineering-Team (Kanban), 10Scap (Scap3-Adoption-Phase1), 10scap2, and 3 others: Deploy jobrunner with scap3 (Trebuchet jobrunner/jobrunner) - https://phabricator.wikimedia.org/T129148#3513276 (10thcipriani) >>! In T129148#3487115, @mmodell wrote: > Do we have a mechanism t... [17:19:57] 10Deployment-Systems, 10Release-Engineering-Team (Kanban), 10Scap (Scap3-Adoption-Phase1), 10scap2, and 3 others: Deploy jobrunner with scap3 (Trebuchet jobrunner/jobrunner) - https://phabricator.wikimedia.org/T129148#3513287 (10thcipriani) [17:22:23] etonkovidova: not that I know of, I would assume someone who is familiar with auth would investigate [17:23:08] greg-g: hmm... I thought it's something infrastructural. [17:23:45] greg-g: sorry, will try to find someone- sigh [17:23:52] well, they'd need to pin point where/what :) [17:35:15] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #62: 04FAILURE in 25 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/62/ [17:43:48] 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review, 10Services (watching), 10User-mobrovac: Scap config management: Jinja2 fills templates with Pythonic values - https://phabricator.wikimedia.org/T145510#3513390 (10thcipriani) a:03thcipriani [18:15:14] Yippee, build fixed! [18:15:15] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #63: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/63/ [18:18:30] 10MediaWiki-Releasing, 10MediaWiki-Vendor: mediawiki/vendor is bloated for tarball releases - https://phabricator.wikimedia.org/T172927#3513508 (10Legoktm) [18:22:04] 10Release-Engineering-Team (Watching / External), 10JobRunner-Service, 10Performance-Team, 10Regression: Investigate 30x increase in Jobrunner errors - https://phabricator.wikimedia.org/T171371#3513524 (10aaron) ``` aaron@terbium:~$ mwscript extensions/WikimediaMaintenance/wmfManageJobs.php aawiki --target... [18:42:38] 10MediaWiki-Releasing, 10MediaWiki-Vendor: mediawiki/vendor is bloated for tarball releases - https://phabricator.wikimedia.org/T172927#3513508 (10Krinkle) It seems we already have REL branches for mediawiki/vendor, but they're currently created by branching off master (which is mostly tailored for WMF). I'd... [18:52:41] 10Continuous-Integration-Infrastructure, 10Technical-Debt: Local Phan errors that don't show up on CI - https://phabricator.wikimedia.org/T172935#3513659 (10WMDE-Fisch) [18:56:10] greg-g: https://phabricator.wikimedia.org/T125976 makes me sad :( So dispatching for wikidata on beta doesn't actually run regularly :( [18:58:55] addshore: ah yes, I forgot about that task/issue. help appreciated? :) [18:59:20] hehe [19:09:49] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #64: 04FAILURE in 20 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/64/ [19:18:02] 10Beta-Cluster-Infrastructure: Run mediawiki::maintenance scripts? - https://phabricator.wikimedia.org/T125976#3513791 (10Addshore) As result of not having this instance for deployment-prep wikidata dispatching doesn't actually run regularly on the beta cluster. Problems with wikidata dispatching have recently b... [19:20:58] 10Beta-Cluster-Infrastructure, 10User-Addshore: Run mediawiki::maintenance scripts? - https://phabricator.wikimedia.org/T125976#3513818 (10Addshore) [19:57:34] Yippee, build fixed! [19:57:35] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #65: 09FIXED in 31 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/65/ [20:05:08] 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review, 10Services (watching), 10User-mobrovac: Scap config management: Jinja2 fills templates with Pythonic values - https://phabricator.wikimedia.org/T145510#3513989 (10thcipriani) [20:06:55] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown, 10MW-1.30-release-notes (WMF-deploy-2017-08-15 (1.30.0-wmf.14)): MediaWiki core was not passing phpcs - https://phabricator.wikimedia.org/T172933#3513994 (10Jdforrester-WMF) p:05Triage>03Normal [20:07:03] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown, 10MW-1.30-release-notes (WMF-deploy-2017-08-15 (1.30.0-wmf.14)): MediaWiki core was not passing phpcs - https://phabricator.wikimedia.org/T172933#3513628 (10Jdforrester-WMF) Now fixed? [20:19:20] Yippee, build fixed! [20:19:20] Project selenium-MinervaNeue-BugFix » chrome,beta,Linux,BrowserTests build #24: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-BugFix/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/24/ [20:29:20] Yippee, build fixed! [20:29:21] Project selenium-MinervaNeue-BugFix » firefox,beta,Linux,BrowserTests build #24: 09FIXED in 30 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue-BugFix/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/24/ [20:43:53] 10MediaWiki-Codesniffer, 10Patch-For-Review: Provide Codesniffer rules to enforce "short" type definitions (int/bool, not integer/boolean) - https://phabricator.wikimedia.org/T145162#3514176 (10Umherirrender) The check is not run for existing @return/@param of private functions, because the sniff is returning... [20:45:23] 10Continuous-Integration-Infrastructure, 10Technical-Debt: Local Phan errors that don't show up on CI - https://phabricator.wikimedia.org/T172935#3514192 (10hashar) @Addshore or @EBernhardson may know. Most probably the errors and warnings have been introduced between 8.0 and 8.5. Surely one could look at b... [20:49:22] 10Continuous-Integration-Infrastructure, 10Technical-Debt: Local Phan errors that don't show up on CI - https://phabricator.wikimedia.org/T172935#3514206 (10Addshore) So yes, we are running 0.8 in CI https://github.com/wikimedia/integration-config/blob/69d77c16a6708def637a210e96d5f28aa44740f2/jjb/mediawiki.yam... [20:55:06] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Technical-Debt: Local Phan errors that don't show up on CI - https://phabricator.wikimedia.org/T172935#3514244 (10WMDE-Fisch) I at least started doing that in ^ [21:07:32] PROBLEM - Puppet errors on deployment-imagescaler02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:17:32] RECOVERY - Puppet errors on deployment-imagescaler02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:23:31] PROBLEM - Puppet errors on deployment-imagescaler02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:32:59] Yippee, build fixed! [22:32:59] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #66: 09FIXED in 19 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/66/