[00:00:50] 3Beta-Cluster: Can not sudo on deployment-cache-mobile03 - https://phabricator.wikimedia.org/T78720#956722 (10greg) p:5Triage>3Normal [00:00:53] 3Beta-Cluster, MediaWiki-extensions-Flow: Beta labs Special:Contributions lags by a long time - https://phabricator.wikimedia.org/T78671#956725 (10greg) p:5Triage>3Normal [00:01:09] 3Beta-Cluster, Release-Engineering, Wikimedia-Logstash: Make logstash in beta public - https://phabricator.wikimedia.org/T76784#956728 (10greg) p:5Triage>3Normal [00:10:00] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [00:25:10] !log restarting jenkins, hope that kicks it enough [00:26:35] annnnd, now jenkins won't come back [00:26:46] Krinkle: around? help please [00:27:00] Krinkle: nvm! [00:27:04] * greg-g was too anxious [00:40:01] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [00:48:03] !log kicking zuul to resolve gearman deadlock [01:12:43] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [01:18:07] greg-g: I was just about to quit but saw the message about Jenkins and Zuul being down. Do you know if that's still an issue? It's usually the gearman thing... [01:20:13] chrismcmahon: yeah, seems it was gearman, I made the wrong move there. [01:20:18] thanks ori [01:33:52] chrismcmahon: can you log what you did [01:35:43] !log clicked "prepare for shutdown", then cancelled operation, hoping to unstick jenkins. beta-scap-eqiad job ran after that. then disabled/enabled gearman [01:35:54] does that work in this channel? [01:37:29] chrismcmahon: it should [01:37:43] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:49:53] * bd808 poked ori to see if he can restart qa-morebots [01:50:31] the morebots project needs a "click here to restart X" web interface [02:21:52] bd808: qdel isn't killing the instance; i still see it in qstat [02:22:04] i'll try qdel -f [02:22:26] hmmm... tool labs is still a mystery to me [02:22:51] and bots are an enigma inside the mystery ;) [02:24:36] tools.morebots@tools-login:~$ qdel -f 4879618 [02:24:36] tools.morebots has registered the job 4879618 for deletion [02:24:36] ............ [02:24:42] tools.morebots@tools-login:~$ qstat | grep qa-logbot [02:24:42] 4879618 0.43058 qa-logbot tools.morebo dr 10/17/2014 20:40:21 continuous@tools-exec-14.eqiad 1 [02:24:42] tools.morebots@tools-login:~$ [02:26:04] i wasn't able to SSH directly into tools-exec-14.eqiad.wmflabs [02:26:47] well at least you tried I guess :( [02:27:11] * bd808 throws rocks at zombie processes in the job grid [02:27:59] doing it as root worked [02:28:08] qdel -f requires privs [02:28:36] god mode unlocked [02:29:01] * bd808 types IDKFA over and over [02:29:13] !log qdel -f'd qa-morebots and started a new instance [02:29:23] Logged the message, Master [02:29:33] w00t [02:29:36] thanks ori [02:29:50] np. gotta run, ttyl. [02:29:59] kthxbye [02:34:53] Project beta-scap-eqiad build #37055: FAILURE in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37055/ [02:45:12] Project beta-scap-eqiad build #37056: STILL FAILING in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37056/ [02:49:34] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<22.22%) [02:50:30] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [02:55:33] Yippee, build fixed! [02:55:34] Project beta-scap-eqiad build #37057: FIXED in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37057/ [03:15:32] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [03:16:38] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [03:35:02] Yippee, build fixed! [03:35:02] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #181: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/181/ [03:40:48] Yippee, build fixed! [03:40:49] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #235: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/235/ [03:41:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [03:49:20] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #460: FAILURE in 31 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/460/ [03:50:29] Yippee, build fixed! [03:50:30] Project browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce build #375: FIXED in 49 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce/375/ [03:51:52] Yippee, build fixed! [03:51:52] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #389: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/389/ [04:03:58] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #289: FAILURE in 9 min 10 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/289/ [04:22:32] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #427: FAILURE in 26 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/427/ [04:32:31] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #270: FAILURE in 8 min 8 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/270/ [06:39:31] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:44:44] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:45:10] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #389: FAILURE in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/389/ [06:50:47] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:01:13] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:09:44] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:43] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:26:12] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [08:11:41] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:36:42] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:54:57] Project beta-scap-eqiad build #37093: FAILURE in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37093/ [08:55:23] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:05:10] Project beta-scap-eqiad build #37094: STILL FAILING in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37094/ [09:15:29] Yippee, build fixed! [09:15:29] Project beta-scap-eqiad build #37095: FIXED in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37095/ [09:17:04] 3Beta-Cluster: VE connection to Parsoid is broken again - https://phabricator.wikimedia.org/T85863#957075 (10hashar) a:3Catrope [09:17:54] 3Beta-Cluster: VE connection to Parsoid is broken again - https://phabricator.wikimedia.org/T85863#957077 (10hashar) 5Open>3Resolved Fixed by Roan who fixed the Parsoid URL on the beta cluster. I have +2ed it and confirmed it to be working by browsing http://en.wikipedia.beta.wmflabs.org/w/index.php?title=... [09:20:22] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:20:46] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:32:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:47:39] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [09:50:44] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [09:51:11] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Can not sudo on deployment-cache-mobile03 - https://phabricator.wikimedia.org/T78720#957143 (10hashar) [09:52:34] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Can not sudo on deployment-cache-mobile03 - https://phabricator.wikimedia.org/T78720#851871 (10hashar) Andrew, Marc, Yuvi, would you mind looking at deployment-cache-mobile03.eqiad.wmflabs please? I can not sudo on it anymore and puppet is broken :-( If it is fixa... [10:02:38] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [10:05:02] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Can not sudo on deployment-cache-mobile03 - https://phabricator.wikimedia.org/T78720#957154 (10Andrew) 5Open>3Resolved a:3Andrew Puppet was choking on an attempt to install sudo-ldap. I do not know why it wasn't installed already, since it comes stock on labs... [10:07:39] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [10:07:55] 3Wikimedia-Labs-Infrastructure, Continuous-Integration: Create labs project for CI disposables instances + OpenStack API credentials - https://phabricator.wikimedia.org/T84988#957158 (10hashar) [10:08:11] 3Continuous-Integration, Wikimedia-Labs-General: setup labs project for continuous integration jobs - https://phabricator.wikimedia.org/T55978#578885 (10hashar) [10:12:55] RECOVERY - Puppet failure on deployment-cache-mobile03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:15:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:18:11] (03CR) 10Hashar: [C: 032] "The composer test entry point has been merged ( https://gerrit.wikimedia.org/r/#/c/182899/ )" [integration/config] - 10https://gerrit.wikimedia.org/r/182900 (owner: 10Hashar) [10:22:20] (03CR) 10Hashar: [C: 032] Switch mediawiki/tools/codesniffer to composer [integration/config] - 10https://gerrit.wikimedia.org/r/182900 (owner: 10Hashar) [10:24:32] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:27:46] (03PS2) 10Hashar: Switch mediawiki/tools/codesniffer to composer [integration/config] - 10https://gerrit.wikimedia.org/r/182900 [10:27:48] (03PS1) 10Hashar: Drop mention of cdb-phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/183010 [10:28:02] (03CR) 10Hashar: [C: 032] Drop mention of cdb-phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/183010 (owner: 10Hashar) [10:28:10] (03CR) 10Hashar: [C: 032] Switch mediawiki/tools/codesniffer to composer [integration/config] - 10https://gerrit.wikimedia.org/r/182900 (owner: 10Hashar) [10:29:03] (03CR) 10jenkins-bot: [V: 04-1] Switch mediawiki/tools/codesniffer to composer [integration/config] - 10https://gerrit.wikimedia.org/r/182900 (owner: 10Hashar) [10:29:16] (03Merged) 10jenkins-bot: Drop mention of cdb-phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/183010 (owner: 10Hashar) [10:35:33] (03Merged) 10jenkins-bot: Switch mediawiki/tools/codesniffer to composer [integration/config] - 10https://gerrit.wikimedia.org/r/182900 (owner: 10Hashar) [10:37:51] (03PS1) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/183013 [10:38:19] (03Abandoned) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/183013 (owner: 10Hashar) [10:45:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [10:54:34] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:50:39] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:15:26] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:20:35] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:22:49] (03PS2) 10Hashar: Clean up {name}-composer comment [integration/config] - 10https://gerrit.wikimedia.org/r/182932 [12:22:57] (03CR) 10Hashar: [C: 032] Clean up {name}-composer comment [integration/config] - 10https://gerrit.wikimedia.org/r/182932 (owner: 10Hashar) [12:23:14] (03PS2) 10Hashar: composer install no more output progress [integration/config] - 10https://gerrit.wikimedia.org/r/182933 [12:24:08] (03CR) 10jenkins-bot: [V: 04-1] Clean up {name}-composer comment [integration/config] - 10https://gerrit.wikimedia.org/r/182932 (owner: 10Hashar) [12:25:56] (03CR) 10jenkins-bot: [V: 04-1] composer install no more output progress [integration/config] - 10https://gerrit.wikimedia.org/r/182933 (owner: 10Hashar) [12:31:45] (03Merged) 10jenkins-bot: Clean up {name}-composer comment [integration/config] - 10https://gerrit.wikimedia.org/r/182932 (owner: 10Hashar) [12:35:47] Project beta-scap-eqiad build #37115: FAILURE in 1 min 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37115/ [12:45:19] Yippee, build fixed! [12:45:20] Project beta-scap-eqiad build #37116: FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37116/ [12:45:37] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [13:17:18] (03CR) 10Hashar: (WIP) gating extensions together (WIP) (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/180494 (owner: 10Hashar) [13:17:27] (03PS4) 10Hashar: (WIP) gating extensions together (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/180494 [13:40:29] 3Continuous-Integration, Release-Engineering: Jenkins: Implement hhvm based voting jobs for mediawiki and extensions (tracking) - https://phabricator.wikimedia.org/T75521#957331 (10hashar) [13:41:35] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:41:57] (03PS3) 10Adrian Lang: Make mwext-WikibaseJavaScriptApi-qunit voting [integration/config] - 10https://gerrit.wikimedia.org/r/180418 [13:42:33] (03PS4) 10Adrian Lang: Make mwext-WikibaseJavaScriptApi-qunit voting [integration/config] - 10https://gerrit.wikimedia.org/r/180418 [13:45:32] (03CR) 10jenkins-bot: [V: 04-1] Make mwext-WikibaseJavaScriptApi-qunit voting [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (owner: 10Adrian Lang) [13:57:45] (03PS5) 10Adrian Lang: Make mwext-WikibaseJavaScriptApi-qunit voting [integration/config] - 10https://gerrit.wikimedia.org/r/180418 [14:06:38] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:50] Project beta-scap-eqiad build #37125: FAILURE in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37125/ [14:28:37] (03CR) 10Hashar: Make mwext-WikibaseJavaScriptApi-qunit voting (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (owner: 10Adrian Lang) [14:29:32] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:35:23] Yippee, build fixed! [14:35:23] Project beta-scap-eqiad build #37126: FIXED in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37126/ [14:47:28] (03PS5) 10Hashar: (WIP) gating extensions together (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/180494 [14:47:50] (03CR) 10Hashar: "Removed Echo due to T78592" [integration/config] - 10https://gerrit.wikimedia.org/r/180494 (owner: 10Hashar) [14:53:34] (03PS6) 10Hashar: (WIP) gating extensions together (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/180494 [14:53:35] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:54:21] (03CR) 10Hashar: "Populate the extensions_load.txt file used to select extensions to be loaded." [integration/config] - 10https://gerrit.wikimedia.org/r/180494 (owner: 10Hashar) [14:54:32] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [15:00:15] (03CR) 10jenkins-bot: [V: 04-1] (WIP) gating extensions together (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/180494 (owner: 10Hashar) [15:00:30] (03CR) 10Hashar: [C: 032] "Updated jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/182933 (owner: 10Hashar) [15:07:54] (03Merged) 10jenkins-bot: composer install no more output progress [integration/config] - 10https://gerrit.wikimedia.org/r/182933 (owner: 10Hashar) [15:13:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:15:11] Project beta-scap-eqiad build #37130: FAILURE in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37130/ [15:15:26] (03PS2) 10Hashar: Add php-composer-validate for wikimedia/wikimania-scholarships [integration/config] - 10https://gerrit.wikimedia.org/r/182769 (owner: 10Legoktm) [15:15:32] (03CR) 10Hashar: [C: 032] Add php-composer-validate for wikimedia/wikimania-scholarships [integration/config] - 10https://gerrit.wikimedia.org/r/182769 (owner: 10Legoktm) [15:16:22] (03Merged) 10jenkins-bot: Add php-composer-validate for wikimedia/wikimania-scholarships [integration/config] - 10https://gerrit.wikimedia.org/r/182769 (owner: 10Legoktm) [15:20:10] grrrit-wm: node version manager? [15:20:18] greg-g: ^ [15:20:23] greg-g: Ah, nevermind. [15:25:02] Project beta-scap-eqiad build #37131: STILL FAILING in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37131/ [15:29:46] (03CR) 10Hashar: Add test to tolerate -composer instead of php-composer-validate (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/180726 (owner: 10Krinkle) [15:31:58] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:34:09] (03PS5) 10Hashar: Adjust test to tolerate -composer job [integration/config] - 10https://gerrit.wikimedia.org/r/180726 (owner: 10Krinkle) [15:35:24] (03CR) 10Hashar: [C: 032] Adjust test to tolerate -composer job [integration/config] - 10https://gerrit.wikimedia.org/r/180726 (owner: 10Krinkle) [15:35:29] Yippee, build fixed! [15:35:29] Project beta-scap-eqiad build #37132: FIXED in 1 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37132/ [15:36:11] (03Merged) 10jenkins-bot: Adjust test to tolerate -composer job [integration/config] - 10https://gerrit.wikimedia.org/r/180726 (owner: 10Krinkle) [15:37:20] 3Beta-Cluster, Multimedia, MediaWiki-extensions-Score: FileBackendException using tag on beta labs: No backend defined with the name 'global-multiwrite' - https://phabricator.wikimedia.org/T85049#957425 (10Gilles) [15:37:40] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:40:42] (03CR) 10Jdlrobson: [C: 04-1] "need to apply changes suggested by Hashar" [integration/config] - 10https://gerrit.wikimedia.org/r/181693 (owner: 10Jdlrobson) [15:41:42] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:43:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [16:01:21] 3Continuous-Integration, Wikimedia-Labs-Infrastructure: CI labs instances can't start on reboot: tmpfs: Bad value 'jenkins-deploy' for mount option 'uid' - https://phabricator.wikimedia.org/T76250#957507 (10hashar) I have also updated lanthanum and gallium so they now have /var/lib/jenkins-slave/tmpfs belonging... [16:02:01] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:02:43] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:06:43] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [17:02:32] twentyafterfour: Reedy meeting ping :) [17:03:08] greg-g: almost there [17:04:46] twentyafterfour: issues? should we start without or are you almost almost almost here? :) [17:09:10] 3Continuous-Integration: Evaluate JClouds Jenkins plugin - https://phabricator.wikimedia.org/T85933#957677 (10hashar) 3NEW [17:09:27] 3Continuous-Integration: Jenkins: Run jobs in disposable VMs - https://phabricator.wikimedia.org/T47499#957684 (10hashar) >>! In T47499#937396, @greg wrote: > (Just came across this, pasting for record keeping: https://wiki.jenkins-ci.org/display/JENKINS/JClouds+Plugin ) I am not sure how fast an instance will... [17:10:04] 3Continuous-Integration: Jenkins: Run jobs in disposable VMs - https://phabricator.wikimedia.org/T47499#957686 (10hashar) [17:10:46] I just had sound issues [17:10:52] but i'm all better now [17:14:41] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:14:51] "Failed to load resource: net::ERR_QUIC_PROTOCOL_ERROR" [17:14:53] This is fun [17:25:00] 3Phabricator, Release-Engineering, § Phabricator-Sprint-Extension: Create a continuous integration plan for Wikimedia Phabricator patches - https://phabricator.wikimedia.org/T85123#957703 (10mmodell) [17:28:41] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:32:37] 3Continuous-Integration, operations: Make sure relevant RelEng people have access to gallium (Chris M, Dan, Mukunda, Zeljko) - https://phabricator.wikimedia.org/T85936#957717 (10greg) 3NEW [17:35:21] 3Phabricator, Release-Engineering, § Phabricator-Sprint-Extension: Create a continuous integration plan for Wikimedia Phabricator patches - https://phabricator.wikimedia.org/T85123#957724 (10mmodell) We discussed this today on the release engineering team meeting. Not much has been decided except that we should... [17:39:38] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:56] 3Continuous-Integration, operations: Make sure relevant RelEng people have access to gallium (Chris M, Dan, Mukunda, Zeljko) - https://phabricator.wikimedia.org/T85936#957731 (10JohnLewis) Mukunda has access to gallium after https://gerrit.wikimedia.org/r/#/c/181211/ @greg So, Chris, Dan and Zeljko need access... [17:42:04] greg-g: ^^ [17:51:12] 3Continuous-Integration, operations: Make sure relevant RelEng people have access to gallium (Chris M, Dan, Mukunda, Zeljko) - https://phabricator.wikimedia.org/T85936#957738 (10greg) >>! In T85936#957731, @JohnLewis wrote: > Mukunda has access to gallium after https://gerrit.wikimedia.org/r/#/c/181211/ > > @gr... [17:57:34] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:58:39] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [18:02:52] greg-g: does Zeljko have shell to the main cluster currently? (just to make sure I'm not missing anything) [18:10:21] 3Continuous-Integration, operations: Make sure relevant RelEng people have access to gallium (Chris M, Dan, Mukunda, Zeljko) - https://phabricator.wikimedia.org/T85936#957750 (10JohnLewis) a:3JohnLewis The above commit gives Chris and Dan access waiting for Greg to formally approve it and then for ops to do th... [18:27:35] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [18:29:21] marxarelli: quick regex q if you have a moment: how to negate a regex? I have e.sub(/\w+$/, '') that does the opposite of what I want. negating the expression e.sub(/!\w+$/, '') gives a type error and e.sub!(/\w+$/, '') isn't what I want either. [18:30:03] chrismcm_: what are you trying to do? [18:31:29] chrismcm_: if you want to just test that a string doesn't match /\w+$/, you would do "something unless str.match(/\w+$/)" [18:31:49] marxarelli: I have a URL that ends in ...wiki/Topic:abc123. I want to capture only the abc123 part. [18:32:27] chrismcm_: if you want to remove any non-word characters (the opposite of \w) at the end of your string, you'd do "str.sub(/\W+$/)" [18:32:32] ah [18:32:59] marxarelli: yeah, I need to remove all characters that precede the ":" and capture the result in a variable [18:33:57] chrismcm_: url.match(/:.*$/) { |m| m[1] } [18:34:03] hm, I have an idea... [18:35:22] chrismcm_: String#match can be given a block that's evaluated if the pattern matches; it's passed the corresponding MatchData object [18:36:04] chrismcm_: it's your go-to for regex captures [18:36:15] oh whoops [18:36:27] the pattern should be: url.match(/:(.*)$/) { |m| m[1] } [18:37:40] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:37:43] closer, not quite there. I'd forgotten about match() in this context [18:38:40] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:39:40] marxarelli: yeah, match() was exactly what I wanted, this should work... [18:42:11] marxarelli: context for this is that a Flow page can have any number of topics with any number of comments, and :index is not adequate to know whether the comment in the test actually belongs to the correct topic [18:42:49] and the only way to tell topics apart is by :id value, which is what I'm capturing here [18:42:56] chrismcm_: got it [18:44:04] chrismcm_: it would be nice if page objects had a concept of scope [18:44:12] it's rather more clever than I like. [18:44:32] chrismcm_: so you could do something like: scope = page.container_element; scope.other_element [18:44:35] marxarelli: they sort of do: [18:44:38] https://github.com/cheezy/page-object/wiki/Nested-Elements [18:44:46] yeah, it's clunky though [18:44:47] but I'm finding some bugs in there using it a lot [18:45:00] it is [18:49:23] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:03:41] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:07:43] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:13:42] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:19:20] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:18] (03CR) 10Legoktm: Adjust test to tolerate -composer job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/180726 (owner: 10Krinkle) [19:24:29] 3Beta-Cluster: VE connection to Parsoid is broken again - https://phabricator.wikimedia.org/T85863#957929 (10Ryasmeen) Verified the fix. Reviewing changes of an edit session, switching to edit source mode while keeping the changes and Saving page working properly now with no error. [19:34:39] (03PS1) 10Legoktm: Use string.endswith() instead of a regex [integration/config] - 10https://gerrit.wikimedia.org/r/183079 [19:35:26] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:35:51] (03CR) 10Legoktm: "Follow up: I2ea544cc44306c78cbaaca38ced093d5a59acb3d" [integration/config] - 10https://gerrit.wikimedia.org/r/180726 (owner: 10Krinkle) [19:38:41] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:47:45] (03CR) 10Krinkle: Adjust test to tolerate -composer job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/180726 (owner: 10Krinkle) [19:48:05] (03CR) 10Krinkle: [C: 032] Use string.endswith() instead of a regex [integration/config] - 10https://gerrit.wikimedia.org/r/183079 (owner: 10Legoktm) [19:48:55] (03Merged) 10jenkins-bot: Use string.endswith() instead of a regex [integration/config] - 10https://gerrit.wikimedia.org/r/183079 (owner: 10Legoktm) [19:50:09] 3Phabricator, Release-Engineering, § Phabricator-Sprint-Extension: Create a continuous integration plan for Wikimedia Phabricator patches - https://phabricator.wikimedia.org/T85123#957995 (10hashar) Apparently we want to run a set of tests when a patch is proposed to the Sprint extension to ensure it is going to... [19:54:37] Yippee, build fixed! [19:54:38] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #271: FIXED in 8 min 44 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/271/ [19:55:09] Project beta-scap-eqiad build #37161: FAILURE in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37161/ [19:56:46] Yippee, build fixed! [19:56:47] Project beta-scap-eqiad build #37162: FIXED in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37162/ [19:57:28] (03PS4) 10Legoktm: Setup php-composer-validate for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/180591 [19:57:37] (03CR) 10Legoktm: "This is ready to be merged now." [integration/config] - 10https://gerrit.wikimedia.org/r/180591 (owner: 10Legoktm) [19:59:18] legoktm: sure thing [20:00:55] (03CR) 10Hashar: [C: 032] "One step forward! Next step would be to migrate the phpunit php lint tests under composer :]" [integration/config] - 10https://gerrit.wikimedia.org/r/180591 (owner: 10Legoktm) [20:02:37] (03Merged) 10jenkins-bot: Setup php-composer-validate for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/180591 (owner: 10Legoktm) [20:04:04] woot :) [20:04:48] 3Continuous-Integration: Convert operations/mediawiki-config to use composer for phpunit and php linting - https://phabricator.wikimedia.org/T85947#958016 (10Legoktm) 3NEW [20:05:23] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [20:21:52] 3MediaWiki-Core-Team, Continuous-Integration, Librarization: Set up composer validate job for operations/mediawiki-config - https://phabricator.wikimedia.org/T76621#958050 (10Legoktm) 5Open>3Resolved I filed T85947 to convert the entire repository to use composer based testing. [20:23:07] hashar: once we merge the all the checks into one -composer job, does that mean non-whitelisted users will no longer have a php linter run? :/ [20:24:38] legoktm: correct [20:24:51] legoktm: until I manage to get some cycles / focus to setup the isolated sandbox on wmflabs [20:31:07] hashar: could we split it into two commands then? composer lint and composer test? [20:32:27] Yippee, build fixed! [20:32:28] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #439: FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/439/ [20:35:12] (03PS1) 10Dduvall: Initialization command for new test suites [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/183089 [20:35:14] (03PS1) 10Dduvall: Improved documentation on EAL [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/183090 [20:36:53] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:55:34] (03PS1) 10Dduvall: Improved documentation on EAL [selenium] - 10https://gerrit.wikimedia.org/r/183092 [20:55:36] (03PS1) 10Dduvall: Merge branch 'env-abstraction-layer' [selenium] - 10https://gerrit.wikimedia.org/r/183093 [20:56:19] (03PS1) 10Hashar: phabricator job to run arc lint on all repo [integration/config] - 10https://gerrit.wikimedia.org/r/183094 [20:57:36] (03CR) 10Hashar: "A very lame first step toward phabricator continuous integration. Example build output:" [integration/config] - 10https://gerrit.wikimedia.org/r/183094 (owner: 10Hashar) [20:59:40] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:01:53] (03PS1) 10Hashar: php lint jobs for some phabricator extensions [integration/config] - 10https://gerrit.wikimedia.org/r/183095 [21:06:05] Yippee, build fixed! [21:06:06] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #379: FIXED in 54 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/379/ [21:06:51] Project browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce build #269: FAILURE in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce/269/ [21:06:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [21:09:46] (03CR) 10Hashar: [C: 032] php lint jobs for some phabricator extensions [integration/config] - 10https://gerrit.wikimedia.org/r/183095 (owner: 10Hashar) [21:16:36] (03Merged) 10jenkins-bot: php lint jobs for some phabricator extensions [integration/config] - 10https://gerrit.wikimedia.org/r/183095 (owner: 10Hashar) [21:20:14] can someone make a jenkins job for https://gerrit.wikimedia.org/r/#/admin/projects/wikidata/gremlin ? Its a standard Maven project like https://gerrit.wikimedia.org/r/#/admin/projects/search/highlighter [21:21:06] manybubbles: hello! Nice to see gremlin coming in :) [21:21:18] its coming! [21:21:20] slowly slowly [21:21:57] manybubbles: the maven job should be easy to define in jjb :] [21:22:04] let me copy paste stuff [21:23:35] manybubbles: what are the goals to use ? [21:23:41] clean package ? [21:24:12] hashar: yeah - that'll do it [21:24:22] in a few days it'll need java 8 unfortunately [21:24:39] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:26:29] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:27:45] (03PS1) 10Hashar: Maven job for wikidata/gremlin [integration/config] - 10https://gerrit.wikimedia.org/r/183129 [21:28:52] manybubbles: doesn't work out of master branch https://integration.wikimedia.org/ci/job/wikidata-gremlin/1/console :( [21:28:56] ERROR: No such file /srv/ssd/jenkins-slave/workspace/wikidata-gremlin/pom.xml [21:29:15] well, I haven't merged that yet [21:29:19] ahh [21:29:33] so have to wait for my config patch to pass test and get merged :] [21:34:13] 3Continuous-Integration: Investigate npm cache-min option to speed up npm install - https://phabricator.wikimedia.org/T85961#958243 (10hashar) 3NEW [21:35:06] (03CR) 10Hashar: [C: 032] Maven job for wikidata/gremlin [integration/config] - 10https://gerrit.wikimedia.org/r/183129 (owner: 10Hashar) [21:35:14] manybubbles: will deploy in roughly 6 minutes [21:35:17] the tests are slow [21:37:38] hashar: hmmm - it still can't find the pom file after I merged the initial one [21:37:44] yeah some oddity [21:37:48] I deleted the workspace manually [21:37:55] the Jenkins git plugin has a bunch of issues [21:37:58] https://integration.wikimedia.org/ci/job/wikidata-gremlin/6/console [21:38:00] SUCCESS [21:38:10] https://integration.wikimedia.org/ci/job/wikidata-gremlin/6/ [21:38:27] you can potentially add some check style / javadoc as well [21:38:35] YuviPanda did so for some of his maven repos [21:39:21] cool - yeah - we can do that once we're ready [21:39:35] hell, we can even add some tests [21:39:37] and potentially publish the resulting doc to doc.wiikimedia.org :] [21:39:44] o really tests ? [21:39:45] what for! [21:40:59] so we should talk about java 8 some time [21:41:09] it'll be required to build this in the next couple days [21:41:09] bring it up on the ops list [21:41:14] its packaged [21:41:17] good [21:41:23] so if it ends up on apt.wikimedia.org [21:41:28] we can get it installed on the CI slaves [21:41:28] its on there [21:41:40] and then in Jenkins figure out the configuration part that make it available [21:41:42] so the trick is that you'll probably still want to use 7 for most other things [21:42:00] so that is what needs to happen - install it and configure it to run for this job [21:42:39] (03Merged) 10jenkins-bot: Maven job for wikidata/gremlin [integration/config] - 10https://gerrit.wikimedia.org/r/183129 (owner: 10Hashar) [21:42:40] Debian alternatives set java to v7 [21:44:00] but I can't find how to add a new JVM in Jenkins [21:44:08] hmmm [21:45:11] hashar: http://stackoverflow.com/questions/19718406/how-many-jvm-invoked-by-jenkins ? [21:46:44] that is interesting [21:47:30] nasty that you'd need a new node [21:47:41] or at least a new "node" from jenkin's perspective [21:47:57] unless you are using a script to kick off the build [21:48:00] the CI job has been merged / deployed so patches proposed to wikidata/gremlin should now trigger the maven job [21:48:44] sweet [21:49:30] AHH [21:49:33] under https://integration.wikimedia.org/ci/configure [21:49:39] there is a setting "JDK" [21:50:01] so we have OpenJdk 6 and 7 [21:50:06] can probably add 8 in there [21:50:12] then configure the Maven job to point to 8 [21:50:59] manybubbles: we will figure it out eventually :] [21:51:04] cool :) [21:52:23] the CI change works https://gerrit.wikimedia.org/r/#/c/183147/ [21:52:24] nice [21:52:37] manybubbles: can you fill a Task to get java8 on the CI Trusty nodes please? [21:53:02] if you could come up with a patch to wikidata/gremlin that requires java8 that would be ideal [21:53:16] (ie fails on java7) [21:55:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:56:32] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:57:45] hashar: will do [21:57:56] got a project for me to stick it in? [21:59:54] why jenkinsbot no submit https://gerrit.wikimedia.org/r/#/c/183147/ ? [22:00:05] oh, permissions [22:00:31] hashar: can you add jenkinsbot as a submitter for that project? I can't see him in the gerrit ui [22:08:10] !log integration-slave1007 chmod -R go+r /srv/deployment/integration/slave-scripts . cscott mentioned build failures of parsoidsvc-jslint which could not read /srv/deployment/integration/slave-scripts/tools/node_modules/jshint/src/cli.js [22:08:16] Logged the message, Master [22:08:59] manybubbles: agh it is in some group [22:09:45] manybubbles: https://gerrit.wikimedia.org/r/#/admin/projects/wikidata/gremlin,access should be good now [22:10:00] thanks! [22:10:03] I can't add him [22:10:09] I tried a few times but he just doesn't show up for me [22:10:37] :-\ [22:12:54] !log integration-slave1005 chmod -R go+r /srv/deployment/integration/slave-scripts [22:12:56] Logged the message, Master [22:13:06] !log jshint complains with: Error: Cannot find module './lib/node' :-( [22:13:09] Logged the message, Master [22:13:44] manybubbles: no clue. I am off to bed though! poke me anytime during your morning [22:13:51] g'night! [22:18:13] Yippee, build fixed! [22:18:13] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #390: FIXED in 25 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/390/ [22:20:44] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [22:22:07] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #384: FAILURE in 14 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/384/ [22:23:52] (03CR) 10Hashar: "I found re.search() to be fine since the composer job will eventually be run with hhvm/zend and thus ends up being suffixed with the php " [integration/config] - 10https://gerrit.wikimedia.org/r/183079 (owner: 10Legoktm) [22:24:00] (03CR) 10Hashar: "Thx :]" [integration/config] - 10https://gerrit.wikimedia.org/r/183079 (owner: 10Legoktm) [22:46:51] hi marxarelli no hurry, but if you have a couple minutes to look this over and see what you think https://gerrit.wikimedia.org/r/#/c/182851/11/tests/browser/features/support/pages/flow_page.rb I think it's the right way to go as far as refactoring for nested elements. [22:48:22] chrismcmahon: cool, i'll have a look [22:49:09] marxarelli: turned out using the :id for the first topic identifier is likely not possible, so it's still index: 0 [22:57:35] 3Continuous-Integration: On integration-slave1007 the directory /srv/deployment/integration/slave-scripts is borked - https://phabricator.wikimedia.org/T85969#958401 (10chasemp) [23:00:29] 3Beta-Cluster, Labs-Team: Setup multimaster salt for large projects using salt-syndic - https://phabricator.wikimedia.org/T78466#958410 (10chasemp) [23:00:36] 3Beta-Cluster, Labs-Team: Setup multimaster salt for large projects using salt-syndic - https://phabricator.wikimedia.org/T78466#958411 (10chasemp) p:5Triage>3Normal [23:19:17] zuul is stuck again? [23:32:56] 3operations, Beta-Cluster: File upload area resorts to 0777 permissions to for uploaded content - https://phabricator.wikimedia.org/T75206#958508 (10chasemp) @ori, you revamped all the apache stuff could you give https://gerrit.wikimedia.org/r/178690 a peek? I'm unsure and this seems like it needs the 'last mi... [23:35:39] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:41:22] Eurgh. Is Zuul/whatever down? Lots of things in postmerge stuck for > 3 hours and a recheck just now didn't cause it to start, well, checking… [23:42:29] Krinkle: Is there a way I can tell? [23:44:45] James_F: Hard to tell because graphs are unreliable. [23:44:53] Krinkle: Yeah. :-( [23:45:00] Someone should look into graphite /zuul analytics weirdness. [23:45:13] Although they look fine now [23:45:44] But still no progress. [23:46:21] James_F: Zuul is not stuck. [23:46:30] PROBLEM - Free space - all mounts on deployment-cache-upload02 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload02.diskspace._srv_vdb.byte_percentfree.value (<100.00%) [23:46:34] James_F: But it ins't processing new events. On purpose it seems. [23:46:37] Krinkle: So what's broken about all the queued items on ? [23:46:46] James_F: I suspect someone did a graceful restart of Zuul 2 hours ago? [23:46:58] and it's still waiting for those jobs to finish [23:47:24] James_F: This build is clogged https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/6722/ [23:47:33] and beta-update doesnt' allow concurrency [23:47:39] so until that one is finished, other ones don't run [23:47:50] Did someone do a schema change? [23:47:52] 22:20:05 Configuration beta-update-databases-eqiad » deployment-bastion-eqiad,kowiki is still in the queue: Waiting for next available executor on deployment-bastion.eqiad [23:50:47] Project beta-update-databases-eqiad build #6722: FAILURE in 1 hr 30 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/6722/ [23:55:42] Project beta-scap-eqiad build #37172: ABORTED in 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/37172/ [23:58:37] marxarelli: I like the inline blocks because it's easier to read and also *much* easier to see the nesting aspect of what these things represent. The previous arrangement was laid out according to "workflow" which doesn't really map to the DOM