[00:13:35] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 13Patch-For-Review, and 4 others: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, RelatedArticles, MobileFront... - https://phabricator.wikimedia.org/T129483#2559577 [00:33:36] 10Continuous-Integration-Infrastructure, 06Labs, 07Wikimedia-Incident: Nodepool instance instance creation quota management - https://phabricator.wikimedia.org/T143016#2559620 (10thcipriani) Some random digging, there is some mention of "quota skew" in the #openstack-infra irc: http://eavesdrop.openstack.org... [01:37:18] 05Gitblit-Deprecate, 13Patch-For-Review: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#2559700 (10Dzahn) @Paladox @Danny_B We got all of the pending changes merged now. Afaict that is all, at least all that was waiting in Gerrit. [01:37:32] 05Gitblit-Deprecate: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#2559701 (10Dzahn) [01:38:15] 05Gitblit-Deprecate: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#2419069 (10Dzahn) Anyone want to run another search across all repos to see if it can be resolved? [01:47:54] 10Continuous-Integration-Config, 06Wikipedia-Android-App-Backlog, 07WorkType-NewFunctionality: Install and use JDK 8 for Android CI testing - https://phabricator.wikimedia.org/T138506#2402389 (10Dzahn) What's up with https://gerrit.wikimedia.org/r/#/c/295880/ ? [01:56:42] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible, 07I18n: On Beta Cluster, MediaWiki namespace override is inconsistently applied - https://phabricator.wikimedia.org/T142863#2559739 (10Mattflaschen-WMF) >>! In T142863#2558004, @greg wrote: >>>! In T142863#2549822, @Mattflaschen-WMF wrote: >> Wasn't... [01:57:15] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible, 07I18n: On Beta Cluster, MediaWiki namespace override is inconsistently applied - https://phabricator.wikimedia.org/T142863#2559740 (10Mattflaschen-WMF) [02:16:48] Project selenium-QuickSurveys » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #121: 04FAILURE in 3 min 47 sec: https://integration.wikimedia.org/ci/job/selenium-QuickSurveys/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/121/ [04:11:58] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T140971#2559813 (10Krinkle) [08:55:31] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 15User-zeljkofilipin: Various browser tests failing due to login error - https://phabricator.wikimedia.org/T142600#2560107 (10zeljkofilipin) [09:45:51] 10Deployment-Systems, 03Scap3: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#2560151 (10fgiunchedi) @thcipriani no worries! I've uploaded the new version now [11:15:29] 06Release-Engineering-Team, 10MediaWiki-General-or-Unknown, 06Operations, 10Traffic, and 5 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#2560365 (10Aklapper) [13:04:21] Project selenium-Math » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #115: 04FAILURE in 20 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/115/ [13:05:58] (03PS8) 10Lethexie: Add usage to forbid superglobals like $_GET,$_POST [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/296395 [13:28:33] !log upgrading elasticsearch to 2.3.4 on deployment-elastic*.deployment-prep + JVM upgrade [13:28:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:10:03] !log upgrading elasticsearch to 2.3.4 on deployment-logstash2.deployment-prep.eqiad.wmflabs [14:10:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:20:10] Yippee, build fixed! [14:20:11] Project selenium-Math » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #116: 09FIXED in 27 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/116/ [14:33:34] Project selenium-WikiLove » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #115: 04FAILURE in 1 min 34 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/115/ [14:58:29] PROBLEM - Puppet run on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:59:15] PROBLEM - Puppet run on deployment-sca01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:02:13] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 13Patch-For-Review, and 4 others: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, RelatedArticles, MobileFront... - https://phabricator.wikimedia.org/T129483#2560821 [15:08:55] PROBLEM - Puppet run on deployment-pdfrender is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [15:13:55] RECOVERY - Puppet run on deployment-pdfrender is OK: OK: Less than 1.00% above the threshold [0.0] [15:14:05] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 07Epic, 05MW-1.28-release-notes, and 3 others: Fix scenarios that fail at en.wikipedia.beta.wmflabs.org or do not run them daily - https://phabricator.wikimedia.org/T94150#2560862 (10Jdlrobson) [15:14:08] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 13Patch-For-Review, and 4 others: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, RelatedArticles, MobileFront... - https://phabricator.wikimedia.org/T129483#2560860 [15:14:57] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 07Epic, 05MW-1.28-release-notes, and 3 others: Fix scenarios that fail at en.wikipedia.beta.wmflabs.org or do not run them daily - https://phabricator.wikimedia.org/T94150#2066052 (10Jdlrobson) [15:14:59] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 13Patch-For-Review, and 4 others: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, RelatedArticles, MobileFront... - https://phabricator.wikimedia.org/T129483#2560876 [15:19:56] PROBLEM - Puppet run on deployment-pdfrender is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:25:46] (03PS6) 10Lethexie: Add detection for calling global functions in target classes. [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/301335 [15:26:34] Just got a segfault on SpamBlacklist's PHP tests. Is this a larger problem or should I just run it again? [15:30:19] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 13Patch-For-Review, and 4 others: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, RelatedArticles, MobileFront... - https://phabricator.wikimedia.org/T129483#2560980 [15:42:38] 05Gitblit-Deprecate: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#2561042 (10Dzahn) Oh, as Danny_B points out of course there is still the link in the task description showing what's left: https://github.com/search?p=2&q=org%3Awikimedia+%22git.wikimedia.org%22&typ... [15:44:00] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 07Epic, 05MW-1.28-release-notes, and 3 others: Fix scenarios that fail at en.wikipedia.beta.wmflabs.org or do not run them daily - https://phabricator.wikimedia.org/T94150#2561046 (10zeljkofilipin) [15:44:02] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 13Patch-For-Review, and 4 others: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, RelatedArticles, MobileFront... - https://phabricator.wikimedia.org/T129483#2561044 [15:54:41] (03CR) 10Zfilipin: "Will do!" [integration/config] - 10https://gerrit.wikimedia.org/r/304740 (https://phabricator.wikimedia.org/T85913) (owner: 10Hashar) [15:57:57] PROBLEM - Puppet run on integration-slave-trusty-1013 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:11:41] * paladox may loose phone connection in Scotland highland lol [16:29:56] RECOVERY - Puppet run on deployment-pdfrender is OK: OK: Less than 1.00% above the threshold [0.0] [16:34:23] I'm entirely clueless where we are at in terms of Differential and Jenkins/CI, but I was told about the Jenkins integration plugin at https://github.com/uber/phabricator-jenkins-plugin this weekend at a conference. [16:34:23] Probably I'm not telling you any news. [16:34:59] 05Gerrit-Migration, 10Differential: Find way to use Differential with plain git (i.e.: without requiring arc) - https://phabricator.wikimedia.org/T127#2561275 (10Aklapper) Only slightly related in terms of convenience and not "solving" this task, Collabora has been maintaining a tool called `git-phab`. See ht... [16:44:46] 10MediaWiki-Releasing, 06Release-Engineering-Team: Include release extensions/skins as submodules of core (maybe vendor too?) - https://phabricator.wikimedia.org/T137564#2561287 (10demon) p:05Normal>03Lowest [17:01:52] twentyafterfour: I think you introduced that bug? :D https://secure.phabricator.com/T11489 [17:02:08] now I'm going to fix it... [17:03:30] andre__: yeah, not new ;) but thanks for thinking of us! [17:07:09] I always think of y'all! :D [17:07:51] (at least I had an interesting conversation how that company uses Phabricator, comparing to us) [17:08:21] I bet [17:19:24] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T140971#2561375 (10Jdforrester-WMF) [17:35:57] PROBLEM - Puppet run on deployment-pdfrender is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:45:11] PROBLEM - Puppet run on deployment-db01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:45:56] RECOVERY - Puppet run on deployment-pdfrender is OK: OK: Less than 1.00% above the threshold [0.0] [18:29:06] PROBLEM - Host deployment-db01 is DOWN: CRITICAL - Host Unreachable (10.68.21.154) [18:39:36] PROBLEM - Puppet run on deployment-db04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:40:54] PROBLEM - Puppet run on deployment-db03 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0] [18:44:35] RECOVERY - Puppet run on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [18:45:55] RECOVERY - Puppet run on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:01:53] PROBLEM - Puppet run on deployment-db03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:10:34] PROBLEM - Puppet run on deployment-db04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:22:48] 06Release-Engineering-Team (Deployment-Blockers), 13Patch-For-Review, 05Release: MW-1.28.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T140971#2561897 (10mmodell) [19:30:24] RECOVERY - Host deployment-parsoid05 is UP: PING OK - Packet loss = 0%, RTA = 0.85 ms [19:31:55] (03PS2) 1020after4: Auto-vote Code-Review+2 on the wikiversions bump [tools/release] - 10https://gerrit.wikimedia.org/r/304604 [19:32:24] 05Gerrit-Migration, 06Release-Engineering-Team, 10releng-201516-q3, 10ArchCom-RfC, and 4 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908#2561906 (10RobLa-WMF) a:05RobLa-WMF>03None Removing myself as assignee to (hopefully) reduce co... [19:32:47] (03CR) 1020after4: "Chad: ok now it votes +2 with the push instead of a separate ssh connection" (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/304604 (owner: 1020after4) [19:36:42] PROBLEM - Host deployment-parsoid05 is DOWN: CRITICAL - Host Unreachable (10.68.16.120) [19:38:15] thcipriani: scap3 + puppet question: is there something I need to setup on a host to allow access by the deploy-service user? Context is deploying striker to californium. [19:38:55] thcipriani: answered my own question by reading scap::target [19:41:46] 06Release-Engineering-Team (Deployment-Blockers), 13Patch-For-Review, 05Release: MW-1.28.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T140971#2561971 (10mmodell) Probably not a blocker but this looks new to me: {T143251} [19:41:53] bd808: heh, that's good :) [19:42:29] and, yeah, mostly just scap::target [19:42:57] a lot of this probably applies for new services https://wikitech.wikimedia.org/wiki/Scap3/Migration_Guide [19:43:06] 06Release-Engineering-Team, 10ArchCom-RfC, 06Developer-Relations, 06WMF-Legal, and 2 others: Create formal process for CREDITS files - https://phabricator.wikimedia.org/T139300#2561974 (10RobLa-WMF) p:05Triage>03High I'm marking this as high priority for my own benefit; not due to urging by ArchCom or... [19:45:51] (03CR) 10Chad: [C: 032] Auto-vote Code-Review+2 on the wikiversions bump [tools/release] - 10https://gerrit.wikimedia.org/r/304604 (owner: 1020after4) [19:46:17] (03Merged) 10jenkins-bot: Auto-vote Code-Review+2 on the wikiversions bump [tools/release] - 10https://gerrit.wikimedia.org/r/304604 (owner: 1020after4) [19:54:26] 03Scap3, 06Community-Tech-Tool-Labs, 06Labs, 10Labs-Infrastructure, 10Striker: Ensure that scap3 from tin can access californium - https://phabricator.wikimedia.org/T143253#2562015 (10bd808) [19:59:05] thcipriani: *nod* I put you on the main puppet patch as a reviewer -- https://gerrit.wikimedia.org/r/#/c/301505/ -- It's working in my Labs test project, but that's no guarantee I actually did the right things :) [20:00:10] heh, I'll take a look :) [20:04:58] 03Scap3, 06Community-Tech-Tool-Labs, 06Labs, 10Labs-Infrastructure, and 2 others: Ensure that scap3 from tin can access californium - https://phabricator.wikimedia.org/T143253#2562039 (10bd808) 05Open>03Resolved a:03bd808 It looks like this should "just work" once all of the puppet rules are in place... [20:17:16] 10Beta-Cluster-Infrastructure, 03Scap3 (Scap3-Adoption-Phase1), 10scap, 10Analytics, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#2562093 (10Milimetric) @elukey is on vacation, and I'm not really sure what changed. But if this is urgent for anyone, just ping me on IRC in #w... [20:36:35] bd808: what realm is californium in for puppet purposes? [20:43:08] Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #119: 04FAILURE in 2 min 7 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/119/ [20:55:34] RECOVERY - Puppet run on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:56:54] RECOVERY - Puppet run on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:15:48] !log starting OCG deploy to beta [21:15:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:21:38] greg-g: I'm trying to put together a plan to deploy Striker (Tool Labs admin console) into prod next week. Security review is done, puppet is pending review, test instance is up and running in a Labs project via puppet. [21:21:58] git-deploy of OCG isn't working on labs [21:22:25] greg-g: Wednesday is looking like it might work for everyone that will be directly involved (me, Yuvi, Jamie). Just wanted to give you a chance to tell me that was horrible [21:22:31] deployment-pdf01.deployment-prep.eqiad.wmflabs: [21:22:31] fetch status: 0 [started: 0 mins ago, last-return: 229 mins ago] [21:23:07] cscott: yuck. any thing better from the verbose status? [21:23:49] that is the verbose status [21:24:39] seems like maybe the salt daemon is just not running? or crashed 230 minutes ago? [21:24:52] non-verbose status is "0/2 minions completed fetch" [21:24:54] * cscott sighs [21:25:36] cscott: I just logged into pdf01. let me see if I can make anything better [21:25:44] sometimes salt jsut gets wacky [21:26:35] !log restarted salt-minion on deployment-pdf01 [21:26:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:26:41] just to dump state: i'm running git-deploy sync from deployment-tin.eqiad.wmflabs from /srv/deployment/ocg/ocg following steps in https://wikitech.wikimedia.org/wiki/OCG#Deploying_the_latest_version_of_OCG [21:27:10] (in case it is not known: CI is backed up again) [21:27:20] cscott: *nod* Let me run the fetch command manually from pdf01. That will give much better error info is something is busted [21:27:29] bd808: now 1/2 minions completed fetch [21:27:45] cscott: was it 01 that completed? [21:27:48] so restarting the salt-minion seems like it did the trick. [21:27:50] [c]ontinue, [y]es, [o]k ? [21:27:50] bd808: yes. [21:28:15] let me slap 02 around then [21:28:15] ori: if you press [c], does it then [c]ancel? ;) [21:28:35] depends on the phase of the moon [21:28:42] !log restarted salt-minion on deployment-pdf02 [21:28:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:28:53] cscott: give is another shot [21:29:24] 2/2 minions completed fetch [21:29:27] let's see if the rest works now. [21:29:35] bam. fuck you salt [21:30:38] 2/2 minions completed checkout as well. so looks like it's working again. thanks bd808 [21:31:04] yw cscott. I have fought with trebuchet many many times [21:31:47] PROBLEM - Puppet run on deployment-aqs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:33:41] !log updated OCG to version e3e0fd015ad8fdbf9da1838c830fe4b075c59a29 [21:33:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:34:39] PROBLEM - Puppet run on integration-slave-precise-1012 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:34:43] blerg. openstack is unhappy with nodepool [21:35:43] PROBLEM - Puppet run on integration-slave-precise-1011 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:36:48] bd808: I don't think it's horrible? What's the failure mode? Nothing else is impacted right (wikitech.wm.o?) [21:37:39] cscott: I hear scap3 is nice :) [21:37:55] greg-g: yes, a migration to service-runner is undoubtedly in the future [21:38:08] we're letting parsoid be the first guinea pig though [21:38:37] * greg-g nods [21:39:16] greg-g: worst cases are, we crash misc varnish, we crash the m5 db shard (labs stuff), we crash californium (horizon). Probable bad is it doesn't work and nobody notices but me. [21:39:27] can someone remind me what the equivalent of graphite.wikimedia.org is in labs? [21:39:41] grepping through the puppet repo isn't jogging my memory [21:40:05] cscott: https://graphite-labs.wikimedia.org/ [21:40:05] https://graphite-labs.wikimedia.org/ [21:40:06] bd808: so any old normal day, sounds like a good day, thinking early morning or? [21:40:52] greg-g: probably SF afternoon actually. Yuvi likes to sleep late :) [21:40:59] oh right [21:41:22] sounds fine at eg 2pm [21:41:25] pacific [21:42:17] *nod* works for me. I'll put it up right after services deploy window [21:42:40] If we bleed into SWAT that shouldn't matter [21:43:07] * greg-g nods [21:45:24] 05Gitblit-Deprecate, 13Patch-For-Review: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#2562418 (10Dzahn) DannyB posted to wikitech-l , asking everybody to chime in and fix the references in their repos https://lists.wikimedia.org/pipermail/wikitech-l/2016-August/... [21:45:57] bd808: I was digging through your striker patch and had a quick question: californium is in the labs support vlan, but still the production realm, correct? [21:46:34] yeah. labs support is prod stuff that does things for labs under the hood [21:47:07] Yuvi and I verified that ssh from tin will work when the hole is opened in ferm by scap::ferm [21:47:09] ack, cool. Just reminding myself how scap::ferm worked and realized it would fail weirdly if the servers were in different realms [21:47:12] cool [21:47:41] one thing that hasn't been checked is the git fetch in the other direction [21:48:04] californium has a public ip, so maybe I better dig up the ferm on that [21:49:00] yarp, good thought. [21:58:29] 05Gitblit-Deprecate, 13Patch-For-Review: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#2562472 (10Paladox) Oh thanks @Dzahn for link and @Danny_B for writing that :) [21:58:52] Yippee, build fixed! [21:58:53] Project selenium-Core » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #123: 09FIXED in 6 min 51 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/123/ [22:12:02] (03PS1) 10Legoktm: Move `rake` jobs off of nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/305408 [22:13:37] Yippee, build fixed! [22:13:38] Project selenium-QuickSurveys » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #122: 09FIXED in 4 min 2 sec: https://integration.wikimedia.org/ci/job/selenium-QuickSurveys/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/122/ [22:15:39] thcipriani: is it alright if I deploy https://gerrit.wikimedia.org/r/#/c/305408/ now? or should I wait? [22:16:16] legoktm: should be fine to deploy [22:17:26] (probably would be good to deploy :)) [22:18:56] 10Continuous-Integration-Infrastructure: Create another jessie slave with 2 executor slots - https://phabricator.wikimedia.org/T142891#2562632 (10Legoktm) Or create two new slaves (medium) with one slot each. [22:19:06] thcipriani: also, who do I need to talk to about creating new permanent slaves ^ [22:27:05] (03CR) 10Legoktm: [C: 032] Move `rake` jobs off of nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/305408 (owner: 10Legoktm) [22:27:19] (03CR) 10Legoktm: [V: 032] Move `rake` jobs off of nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/305408 (owner: 10Legoktm) [22:27:57] !log deploying https://gerrit.wikimedia.org/r/305408 [22:28:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:29:00] aw crap [22:29:02] that's broken [22:29:32] Project selenium-QuickSurveys » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #124: 04FAILURE in 1 min 26 sec: https://integration.wikimedia.org/ci/job/selenium-QuickSurveys/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/124/ [22:30:47] (03PS1) 10Legoktm: rake: Fix bundle install path [integration/config] - 10https://gerrit.wikimedia.org/r/305411 [22:31:04] (03CR) 10Legoktm: [C: 032] "Already deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/305411 (owner: 10Legoktm) [22:31:04] legoktm: Aye, rake builds failing for mw core commits [22:31:12] just fixed it [22:32:37] https://integration.wikimedia.org/ci/job/rake/9/console [22:32:39] looks good [22:32:43] (03Merged) 10jenkins-bot: rake: Fix bundle install path [integration/config] - 10https://gerrit.wikimedia.org/r/305411 (owner: 10Legoktm) [22:33:15] sorry about that [22:34:54] legoktm: we don't have an I broke CI shirt yet. :) [22:35:14] heh [22:36:28] (03CR) 1020after4: [C: 032] Use composer in DonationInterface hhvm tests [integration/config] - 10https://gerrit.wikimedia.org/r/301025 (https://phabricator.wikimedia.org/T141309) (owner: 10Awight) [22:36:56] (03CR) 1020after4: "awight: Sorry I just now saw this, I would have merged sooner if I had noticed." [integration/config] - 10https://gerrit.wikimedia.org/r/301025 (https://phabricator.wikimedia.org/T141309) (owner: 10Awight) [22:41:17] legoktm: I can look into adding more jessie instances. Thank you for your help with CI now that nodepool has started exploding/falling over, it's been hairy :\ [22:41:54] thanks and np :) [22:42:41] thcipriani: I have to step away here, I can revert changes or leave it w/ puppet disabled [22:42:53] I think we should leave it atm [22:43:15] chasemp: yeah, I'd say leave it for the time being. [22:43:25] it's seemed more stable over the short term than it ahs [22:43:27] *has [22:44:32] I see jenkins responses for CI around 1-3m per changeset and it seems stable yeah [22:45:10] that said thcipriani any idea why this is failing? https://gerrit.wikimedia.org/r/#/c/305401/ [22:45:19] * thcipriani looks [22:45:25] is there an easy way to filter to events happening in CI selenium web requests in logstash? [22:46:28] chasemp: can you update the task with what you did? [22:47:24] 10Continuous-Integration-Config: Move npm-node-4 jobs off of nodepool - https://phabricator.wikimedia.org/T142892#2562753 (10Legoktm) My gut feeling based off of staring at the zuul status/queues page for a few days during peak time is that we should just do this now, but leave the mediawiki core and oojs-ui job... [22:48:20] greg-g: yes I will [22:48:24] chasemp: ty good sir [22:48:29] want to see it hold for awhile still [22:48:36] chasemp: I'm mostly thining of antoine tryign to catch up next week [22:48:46] * greg-g nods [22:48:53] chasemp: commented [22:49:35] thanks legoktm [22:50:02] that test said 'ERROR: InvocationError: '/mnt/jenkins-workspace/workspace/operations-puppet-tox/.tox/pep8/bin/flake8'' [22:50:04] chasemp: modules/toollabs/files/monitoring/sge.py:49:43: E127 continuation line over-indented for visual indent [22:50:11] which I took to be...a failure to run teh commands to lint itself [22:50:37] oh, heh, beat me to it [22:50:42] tox output can be a bit confusing [22:50:55] mostly because it bolds and highlights everything but the actual error output [22:51:05] heh [22:57:17] (03CR) 10Awight: "Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/301025 (https://phabricator.wikimedia.org/T141309) (owner: 10Awight) [23:03:23] (03PS7) 10Awight: Use composer in DonationInterface hhvm tests [integration/config] - 10https://gerrit.wikimedia.org/r/301025 (https://phabricator.wikimedia.org/T141309) [23:11:26] how do I tell which wiki https://integration.wikimedia.org/ci/view/Selenium/job/selenium-QuickSurveys/124/console has been running on? it has MEDIAWIKI_ENVIRONMENT=beta so I would assume beta cluster but I can't find any relevant logs [23:13:06] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 03Fundraising Sprint Octopus Untangling, and 3 others: Continuous integration: DonationInterface needs composer variant - https://phabricator.wikimedia.org/T141309#2562903 (10DStrine) [23:17:20] I have yet to figure out how to find the logs from the matrix jobs [23:46:44] 06Release-Engineering-Team, 10Phabricator: Search not finding task - https://phabricator.wikimedia.org/T143014#2563033 (10Dzahn) [23:47:53] 06Release-Engineering-Team, 10Phabricator: Search not finding task - https://phabricator.wikimedia.org/T143014#2554112 (10Dzahn) Since it's about Phabricator search i replaced Operations with Release Engineering. Tell me if that's wrong and you need Operations to run something. [23:58:43] PROBLEM - Puppet run on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]