[00:13:09] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:17:29] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 47405 bytes in 0.622 second response time [01:03:41] 10Continuous-Integration-Infrastructure, 6Collaboration-Team, 10Flow: Database field 'workflow.workflow_wiki' too short (not compatible with mediawiki's iw_wikiid) - https://phabricator.wikimedia.org/T93463#1263616 (10Springle) [01:05:30] 10Beta-Cluster, 6Release-Engineering, 10Continuous-Integration-Config, 10Parsoid: Parsoid patches don't update Beta Cluster automatically -- only deploy repo patches seem to update that code - https://phabricator.wikimedia.org/T92871#1263629 (10ssastry) Both solutions are messy since npm install can instal... [02:10:56] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:20:51] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 29100 bytes in 0.762 second response time [05:06:47] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:28:37] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:29:20] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:32:40] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:33:28] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 47405 bytes in 0.571 second response time [05:33:58] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 46886 bytes in 0.799 second response time [05:37:32] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 46885 bytes in 0.813 second response time [05:43:57] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:51] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 29100 bytes in 0.830 second response time [06:44:45] 10Beta-Cluster, 6Labs: Move logs off NFS on beta - https://phabricator.wikimedia.org/T98289#1263912 (10yuvipanda) 3NEW [07:02:15] 6Release-Engineering, 10MediaWiki-Debug-Logging, 6Security-Team, 6operations, 10procurement: Store unsampled API and XFF logs - https://phabricator.wikimedia.org/T88393#1263932 (10fgiunchedi) [07:40:13] (03CR) 1020after4: [C: 031] Update statsd events [tools/scap] - 10https://gerrit.wikimedia.org/r/208987 (https://phabricator.wikimedia.org/T64667) (owner: 10BryanDavis) [07:49:58] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:54:51] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 29100 bytes in 0.725 second response time [08:05:14] 10Continuous-Integration-Infrastructure: Bump python-gear package to 0.5.6 - https://phabricator.wikimedia.org/T98294#1264016 (10hashar) 3NEW a:3hashar [08:05:27] 10Continuous-Integration-Infrastructure, 7Zuul: Bump python-gear package to 0.5.6 - https://phabricator.wikimedia.org/T98294#1264024 (10hashar) [08:07:13] 5Continuous-Integration-Isolation, 7Nodepool: Bump Nodepool package to v0.1.0 and propose it to Debian - https://phabricator.wikimedia.org/T98295#1264026 (10hashar) 3NEW [08:38:38] 10Beta-Cluster, 10MediaWiki-extensions-GWToolset, 6Multimedia, 7HHVM, 5Patch-For-Review: GWToolset XML upload fails with “The file that was uploaded exceeds the upload_max_filesize and/or the post_max_size directive in php.ini” on hhvm 3.6 - https://phabricator.wikimedia.org/T97415#1264053 (10Jason.nlw)... [08:48:47] 5Continuous-Integration-Isolation, 6Labs, 10Labs-Infrastructure: Include Base::Standard-packages in labs images - https://phabricator.wikimedia.org/T94995#1264068 (10hashar) When an instance boot (Trusty in this case: Notice: Finished catalog run in 56.05 seconds Cloud-init v. 0.7.5 running 'modules... [09:12:14] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:16:25] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 47405 bytes in 0.565 second response time [09:44:17] 10Beta-Cluster: test.wikipedia.beta.wmflabs.org redirects to http://test.wikimedia.beta.wmflabs.org/wiki/Main_Page - https://phabricator.wikimedia.org/T97489#1264188 (10hashar) The beta cluster has a `testwiki` database. The router recognize the wikiPedia and wikiMedia flavors. What is missing is the mobile ver... [09:44:27] 10Beta-Cluster: test.wikipedia.beta.wmflabs.org redirects to http://test.wikimedia.beta.wmflabs.org/wiki/Main_Page - https://phabricator.wikimedia.org/T97489#1264189 (10hashar) p:5Triage>3Normal [09:59:38] 10Browser-Tests: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#1264263 (10zeljkofilipin) 5declined>3Open a:5hashar>3None [10:00:10] 10Browser-Tests: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#23427 (10zeljkofilipin) I still think this should be done (removing RVM comments from Gemfile). [10:06:18] 10Browser-Tests, 6Release-Engineering: Determine weekly triage meeting for Browser Tests - https://phabricator.wikimedia.org/T98207#1264310 (10zeljkofilipin) I think both @dduvall and me can lead the meeting. Since the two of us are 9 hours away, early in his day and late in my day would be a good time. Dan,... [10:45:50] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [11:21:09] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:59] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 46893 bytes in 0.892 second response time [12:17:41] 10Beta-Cluster, 10MediaWiki-extensions-GWToolset, 6Multimedia, 7HHVM, 5Patch-For-Review: GWToolset XML upload fails with “The file that was uploaded exceeds the upload_max_filesize and/or the post_max_size directive in php.ini” on hhvm 3.6 - https://phabricator.wikimedia.org/T97415#1264578 (10Bawolff) O... [12:50:56] 10Continuous-Integration-Infrastructure, 7Zuul: Bump python-gear package to 0.5.6 - https://phabricator.wikimedia.org/T98294#1264693 (10hashar) p:5Triage>3Low [13:36:32] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:44:37] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<11.11%) [13:49:35] 10Browser-Tests, 6Release-Engineering: Determine weekly triage meeting for Browser Tests - https://phabricator.wikimedia.org/T98207#1264765 (10zeljkofilipin) Since Dan and I already have a weekly pairing session, we might do triage every other week, or set up a separate meeting for triage, if we manage to find... [13:57:15] twentyafterfour: about? [14:01:30] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [14:14:38] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [14:20:50] 10Beta-Cluster, 6Labs: Move logs off NFS on beta - https://phabricator.wikimedia.org/T98289#1264824 (10hashar) There are several topics: syslog -------- The instances have rsyslog configured to relay logs to deployment-bastion.eqiad.wmflabs The deployment-bastion is setup with `role::syslog::centralserver`... [14:21:00] hello [14:21:20] how can I run the browser tests in saucelabs? [14:24:16] stephanebisson: https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Running_tests#Running_browser_tests_at_Sauce_Labs might be up to date :) [14:24:31] you need to pass env variables SAUCE_ONDEMAND_USERNAME and SAUCE_ONDEMAND_ACCESS_KEY [14:24:49] and the browser tests will make use of SauceLabs magically [14:25:41] hashar: Thanks! Any idea where I can find a user/key? [14:26:53] chasemp: here [14:27:50] 10Browser-Tests: mediawiki_selenium should document SauceLabs usage - https://phabricator.wikimedia.org/T98331#1264867 (10hashar) 3NEW [14:28:07] 10Browser-Tests, 7Documentation: mediawiki_selenium should document SauceLabs usage - https://phabricator.wikimedia.org/T98331#1264874 (10hashar) [14:28:12] stephanebisson: and I have filled https://phabricator.wikimedia.org/T98331 :) [14:28:29] stephanebisson: we have a wikimedia generic account and can create sub accounts from it [14:29:06] stephanebisson: beside that I don't know that much :/ [14:29:47] hashar: who has the power to create such sub accounts? [14:29:53] stephanebisson: ah https://saucelabs.com/signup [14:30:11] stephanebisson: and open a free account maybe. Then fill a task for zeljkof-away he hold the credentials [14:30:30] I am not familiar with it to be honest [14:30:44] hashar: sounds good, thanks! [14:31:59] stephanebisson: definitely fill a task or mail zeljkof about it though [14:43:41] 10Beta-Cluster, 6Labs: Move logs off NFS on beta - https://phabricator.wikimedia.org/T98289#1264931 (10bd808) It should be pretty easy to get all of the logging in the beta cluster to flow into logstash. Apache and HHVM should be configured to log to syslog and then rsyslog rules used to forward to Logstash. T... [15:10:26] (03CR) 10Chad: [C: 032] Update statsd events [tools/scap] - 10https://gerrit.wikimedia.org/r/208987 (https://phabricator.wikimedia.org/T64667) (owner: 10BryanDavis) [15:10:49] (03Merged) 10jenkins-bot: Update statsd events [tools/scap] - 10https://gerrit.wikimedia.org/r/208987 (https://phabricator.wikimedia.org/T64667) (owner: 10BryanDavis) [15:12:02] greg-g: good morning :} [15:13:23] !log Updated scap to 57036d2 (Update statsd events) [15:13:29] Logged the message, Master [15:13:38] hashar: 'ello! [15:29:45] 10Beta-Cluster, 10Continuous-Integration-Config: beta-update-databases-eqiad should get list of DB to update from mediawiki-config all-labs.dblist - https://phabricator.wikimedia.org/T98342#1265105 (10hashar) 3NEW [15:31:44] 10Beta-Cluster, 7Blocked-on-RelEng, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, and 3 others: Setup new wikis in Beta Cluster for Content Translation - https://phabricator.wikimedia.org/T90683#1265119 (10hashar) The Jenkins job that update the databases can probably read th... [16:29:06] PROBLEM - Puppet failure on deployment-zotero01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:34:45] 10Continuous-Integration-Infrastructure, 6operations, 7Blocked-on-Operations: Build Debian package ruby-jsduck for Jessie - https://phabricator.wikimedia.org/T95008#1265396 (10hashar) The packages are already build for Trusty but I don't think they will work as is on Jessie (ex: different ruby version). The... [16:38:21] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 6Scrum-of-Scrums, 6operations, and 2 others: Jenkins: Re-enable lint checks for Apache config in operations-puppet - https://phabricator.wikimedia.org/T72068#1265437 (10hashar) We talked about this task during our weekly RelEng checkin. The Jen... [17:04:48] stephanebisson: ping me if you need help [17:06:06] zeljkof: I'm all set for now with a free saucelabs account (2h limit) [17:06:27] stephanebisson: if you need moar, let me know [17:06:39] zeljkof: well, I need more :) [17:07:07] stephanebisson: I am in a meeting now, can you create a phab ticket and assign it to me? [17:07:12] so I do not forget [17:07:17] zeljkof: I'm trying to figure out why our tests fail a lot more in saucelabs than locally [17:07:49] stephanebisson: also, feel free to ping me if you would like to pair on it [17:08:27] zeljkof: https://phabricator.wikimedia.org/T98336 [17:09:48] zeljkof: sure, thanks! [17:10:07] stephanebisson: will do it after the meeting (if I do not forget) [17:10:12] also, take a look at https://office.wikimedia.org/wiki/Selenium_passwords [17:10:35] wikimedia-jenkins user has unlimited minutes [17:11:22] zeljkof: can I just use wikimedia-jenkins or is it better if I have my own account? [17:11:37] stephanebisson: what ever you prefer, both would work [17:12:19] zeljkof: if it's the same price I would prefer my own account so the dashboard is less crowded [17:12:31] stephanebisson: same price for us :) [17:14:43] 10Beta-Cluster, 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Add niedzielski releasers-mobile in production and deployment-prep in labs - https://phabricator.wikimedia.org/T98179#1265666 (10Dzahn) approval (and original request to do this before that) was on https://phabricator.wikimedia.org/T97... [17:17:03] 10Beta-Cluster, 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Add niedzielski releasers-mobile in production and deployment-prep in labs - https://phabricator.wikimedia.org/T98179#1265673 (10Dzahn) 5Open>3Resolved a:3Dzahn @niedzielski and this one gave you access to "caesium" to be able to... [17:38:59] 10Beta-Cluster, 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Add niedzielski releasers-mobile in production and deployment-prep in labs - https://phabricator.wikimedia.org/T98179#1265848 (10Niedzielski) I'm in! Thanks (and thanks for the proxy note)! [17:50:20] 10Browser-Tests: Create a Saucelabs sub account for sbisson - https://phabricator.wikimedia.org/T98336#1265876 (10zeljkofilipin) 5Open>3stalled p:5Triage>3Normal [17:50:30] 10Browser-Tests: Create a Saucelabs sub account for sbisson - https://phabricator.wikimedia.org/T98336#1264906 (10zeljkofilipin) 5stalled>3Resolved [17:50:52] 10Browser-Tests: Create a Saucelabs sub account for sbisson - https://phabricator.wikimedia.org/T98336#1264906 (10zeljkofilipin) Sent invitation to @sbisson. Let me know if you have problems creating the account. [18:21:31] PROBLEM - Puppet failure on integration-zuul-packaged is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:51:23] 10Browser-Tests, 6Release-Engineering: Determine weekly triage meeting for Browser Tests - https://phabricator.wikimedia.org/T98207#1266179 (10dduvall) I could probably do 8:30am, either on Wednesdays before SoS or on Thursdays before our regular pairing sessions. [19:03:54] 10Browser-Tests: Create a Saucelabs sub account for sbisson - https://phabricator.wikimedia.org/T98336#1266222 (10SBisson) All good. Thanks. [20:18:24] (03CR) 10Legoktm: [C: 032] Add script that checks for incorrectly capitalized class names [tools/code-utils] - 10https://gerrit.wikimedia.org/r/207720 (owner: 10PleaseStand) [20:27:21] (03PS1) 10Chad: make-release: use https by default instead of ssh [tools/release] - 10https://gerrit.wikimedia.org/r/209368 [20:27:36] <^d> legoktm: If you don't mind ^ [20:27:58] (03CR) 10Legoktm: [C: 032] make-release: use https by default instead of ssh [tools/release] - 10https://gerrit.wikimedia.org/r/209368 (owner: 10Chad) [20:28:06] <^d> ty [20:29:07] ^d: also https://gerrit.wikimedia.org/r/#/q/status:open+project:mediawiki/tools/release+owner:%22Legoktm+%253Clegoktm.wikipedia%2540gmail.com%253E%22,n,z :) [20:29:53] <^d> Yeah I had looked [20:30:10] <^d> I can merge after I finish running today [20:31:34] (03Merged) 10jenkins-bot: Add script that checks for incorrectly capitalized class names [tools/code-utils] - 10https://gerrit.wikimedia.org/r/207720 (owner: 10PleaseStand) [20:31:44] 10Browser-Tests, 10Continuous-Integration-Infrastructure, 7Tracking: Fix or delete browsertests* Jenkins jobs that are failing for more than a week (tracking) - https://phabricator.wikimedia.org/T94150#1266633 (10DannyH) [20:37:45] (03Merged) 10jenkins-bot: make-release: use https by default instead of ssh [tools/release] - 10https://gerrit.wikimedia.org/r/209368 (owner: 10Chad) [21:17:25] PROBLEM - Puppet staleness on deployment-eventlogging02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [22:10:06] 10Browser-Tests, 6Release-Engineering: Determine weekly triage meeting for Browser Tests - https://phabricator.wikimedia.org/T98207#1267150 (10greg)