[00:19:16] (03CR) 10Samwilson: "Is there a Phabricator ticket for this too? Most of your commit message should be in Phabricator, and the message here just relating to th" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/333123 (owner: 10Jeroen De Dauw) [00:26:11] 10Continuous-Integration-Config, 10VisualEditor: Some CI jobs are failing because node is now on node 6 ("Assertion error: node version v6.9.1 does not match '^v4[.]'") - https://phabricator.wikimedia.org/T155685#2954863 (10Ricordisamoa) [00:54:49] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team: Browser test Jenkins videos are corrupt - https://phabricator.wikimedia.org/T155794#2954941 (10Jdlrobson) [01:25:23] 10MediaWiki-Codesniffer: Make a contributing.md file or add to README, steps to contribute to Mediawiki Codesniffer - https://phabricator.wikimedia.org/T133478#2232861 (10Samwilson) I think this sort of documentation should be at https://www.mediawiki.org/wiki/Manual:PHP_Codesniffer (and not be WMF-specific). At... [04:07:26] Yippee, build fixed! [04:07:27] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #271: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/271/ [06:30:06] (03CR) 10Hashar: [C: 031] Do not require specific versions of PHPCS [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/333123 (owner: 10Jeroen De Dauw) [06:31:53] (03CR) 10Samwilson: [C: 031] Do not require specific versions of PHPCS [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/333123 (owner: 10Jeroen De Dauw) [06:36:40] 10Gerrit, 07Documentation: Enable and document "WIP" workflow status in Gerrit - https://phabricator.wikimedia.org/T135245#2955267 (10hashar) [06:36:42] 10Gerrit, 10Developer-Wishlist (2017), 13Patch-For-Review: Allow Gerrit changes to be properly marked as WIP - https://phabricator.wikimedia.org/T155530#2955269 (10hashar) [06:37:32] 10Gerrit, 10Developer-Wishlist (2017), 13Patch-For-Review: Allow Gerrit changes to be properly marked as WIP - https://phabricator.wikimedia.org/T155530#2946281 (10hashar) Following the wikitech-l thread you started in September 2015, we had T135245 filled. So marking as a dupe. [06:41:54] Project selenium-Wikibase » chrome,test,Linux,contintLabsSlave && UbuntuTrusty build #244: 04FAILURE in 2 hr 0 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/244/ [06:59:21] Project selenium-Wikibase » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #244: 04FAILURE in 2 hr 18 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/244/ [08:45:43] Project beta-code-update-eqiad build #139560: 04FAILURE in 1 min 44 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/139560/ [08:51:07] Jenkins is broken on master (again) It looks definitely an infra issue this time: https://gerrit.wikimedia.org/r/333203 Can't connect to mysql [08:54:17] 10Continuous-Integration-Infrastructure, 07Jenkins, 03Wikidata-Sprint: Jenkins on Wikibase: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' - https://phabricator.wikimedia.org/T155815#2955413 (10Ladsgroup) [08:54:29] 10Continuous-Integration-Infrastructure, 07Jenkins, 03Wikidata-Sprint: Jenkins on Wikibase: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' - https://phabricator.wikimedia.org/T155815#2955427 (10Ladsgroup) p:05Triage>03Unbreak! [08:54:37] ^ [08:55:45] Project beta-code-update-eqiad build #139561: 04STILL FAILING in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/139561/ [09:04:14] Project beta-code-update-eqiad build #139562: 04STILL FAILING in 1 min 39 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/139562/ [09:04:50] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: mysql does not start when Trusty instances spawn - https://phabricator.wikimedia.org/T141450#2955439 (10hashar) [09:04:52] 10Continuous-Integration-Infrastructure, 07Jenkins, 03Wikidata-Sprint: Jenkins on Wikibase: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' - https://phabricator.wikimedia.org/T155815#2955441 (10hashar) [09:05:21] !log integration restarted mysql on trusty permanent slaves T141450 T155815 salt -v '*trusty*' cmd.run 'service mysql start' [09:05:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:05:26] T155815: Jenkins on Wikibase: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' - https://phabricator.wikimedia.org/T155815 [09:05:26] T141450: mysql does not start when Trusty instances spawn - https://phabricator.wikimedia.org/T141450 [09:05:35] Amir1: ^^^^:) [09:05:37] Amir1: mysql is back [09:06:00] Project beta-code-update-eqiad build #139563: 04STILL FAILING in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/139563/ [09:06:07] thanks hashar \o/ [09:09:41] Project beta-code-update-eqiad build #139564: 04STILL FAILING in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/139564/ [09:11:20] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team: Browser test Jenkins videos are corrupt - https://phabricator.wikimedia.org/T155794#2955468 (10zeljkofilipin) a:03zeljkofilipin [09:13:26] beta-code-update-eqiad is me testing [09:15:00] Project beta-code-update-eqiad build #139565: 04STILL FAILING in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/139565/ [09:16:46] Project beta-code-update-eqiad build #139566: 04STILL FAILING in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/139566/ [09:18:28] !log beta: reset workspace of /srv/mediawiki-staging/php-master/extensions/reCaptcha it had a .gitignore local hack for some reason [09:18:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:25:50] Project beta-code-update-eqiad build #139567: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/139567/ [09:35:48] Yippee, build fixed! [09:35:48] Project beta-code-update-eqiad build #139568: 09FIXED in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/139568/ [09:56:20] 10Gerrit, 10DBA, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2955552 (10Marostegui) >>! In T145885#2953741, @demon wrote: >>>! In T145885#2951958, @Marostegui wrote: >> That will convert ALL... [10:08:02] 10Gerrit, 06Release-Engineering-Team, 10DBA, 06Operations, 13Patch-For-Review: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2955567 (10Marostegui) See: T145885#2951958 This change requires us to change 2 global flags on the host. As well as doing the AL... [10:13:47] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Puppet fails on integration instances: nfs_mount[home-on-labstoresvc]: umount: /home: not mounted - https://phabricator.wikimedia.org/T155820#2955587 (10hashar) [10:14:13] !log puppet fails on "integration" labs instances due to an attempt to unmount the non existing NFS /home. Filled T155820 [10:14:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:14:17] T155820: Puppet fails on integration instances: nfs_mount[home-on-labstoresvc]: umount: /home: not mounted - https://phabricator.wikimedia.org/T155820 [10:14:47] hashar: is that running off it's own puppetmaster or anything? [10:14:55] afaik madhu knew about and fixed that [10:15:10] chasemp: yeah I haven't dig in the puppet log yet :} [10:15:24] hashar: what instance failed this way? [10:15:35] chasemp: do ignore it! it must be late for you right now. I will figure it out :} [10:15:43] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Puppet fails on integration instances: nfs_mount[home-on-labstoresvc]: umount: /home: not mounted - https://phabricator.wikimedia.org/T155820#2955587 (10yuvipanda) Is this running the latest puppet code? [10:16:05] yuvipanda: yeah got triggered when I rebased the puppet repo yesterday [10:16:18] hashar: it was fixed in a later commit [10:16:21] I'm all the way back around and up again :) [10:16:32] but yeah it does look like that shouldn't happen at HEAD [10:16:39] looking [10:16:43] if you could update and validate that would be good [10:16:58] it is up-to-date [10:17:13] but when I rebased yesterday it was lagging out by like 2 or 3 days [10:17:15] can you create a new instance inthis project now? [10:17:56] yup [10:18:40] we know this was teh case briefly for a period day before yesterday fyi [10:19:51] it is only a few instances being impacted [10:20:00] the bulk of them are all fine at least [10:20:08] my main question is, are new instances? [10:20:45] hashar-nfs.integration.eqiad.wmflabs is booting [10:21:07] chasemp: but really you should sleep ;-} [10:21:12] it is not the end of the world! [10:21:13] so my understanding is that integration shouldn't be getting any NFS mounts [10:21:34] correct yuvipanda [10:21:47] I think it was one of the first project to phase out usage of NFS entirely [10:22:04] cause whenever NFS had an issue the whole CI stalled [10:22:38] I haven't dig in puppet code yet to figure out why cleanup-/home is triggered [10:23:23] yes [10:24:15] running first puppet on hashar-nfs.integration.eqiad.wmflabs [10:25:04] looks fine [10:27:15] hashar: :D oK! [10:28:01] hashar: there is a chance some instances caught side effects during the course of that fixup, but if all is well on a fresh instance that makes sense. If you have a few that are in an odd state we can look at them if you put up a list somewhere [10:28:08] or just make new ones idk what's easier [10:28:33] I see [10:28:33] Error: /Stage[main]/Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]: Could not evaluate: Could not retrieve information from environment production source(s) file:/var/lib/puppet/client/ssl/certs/ca.pem [10:28:34] there though [10:28:41] which is ...interesting [10:29:06] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Puppet fails on integration instances: nfs_mount[home-on-labstoresvc]: umount: /home: not mounted - https://phabricator.wikimedia.org/T155820#2955638 (10hashar) The puppetmaster was stall with 2-3 days of lag and I rebased yesterday just... [10:29:26] chasemp: yeah that is the case on new instances [10:29:34] there is a task for it somewhere [10:29:47] :D fair enough [10:30:07] looks like the borked instance have a hiera( mount_nfs, true) that yields true [10:30:10] so either mount_nfs is true [10:30:14] or not set and default to true [10:30:14] ah [10:30:15] bah :( [10:30:24] that would have caught them up in the storm [10:30:41] iirc you can set a project wide mount_nfs: false [10:31:00] but I guess that the node explicit would win anyhow [10:31:06] so yeah, hiera cleanup fest :) [10:31:53] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Puppet fails on integration instances: nfs_mount[home-on-labstoresvc]: umount: /home: not mounted - https://phabricator.wikimedia.org/T155820#2955641 (10yuvipanda) Yes, there's probably going to be a refactor at some point for that. Doe... [10:33:51] yuvipanda: have you changed something ? [10:35:11] hashar: yes I deleted the mount_nfs bits from hiera since the configuration they were in hasn't been doing anything for a long time [10:35:38] so I guess that should be replaced by mount_nfs: true [10:35:40] err [10:35:42] mount_nfs: false [10:36:19] or in the puppet nfs-mounts.yaml file [10:36:58] PROBLEM - Puppet run on hashar-nfs is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:37:48] restarted puppetmaster, running puppet on integration-slave-jessie1001.integration.eqiad.wmflabs [10:38:22] still no luck :D [10:38:44] hashar: yes, and nfs-mounts.yaml already has proper config for integration (aka no config) [10:46:13] $ /usr/local/sbin/nfs-mount-manager check [10:46:13] Usage: /bin/grep [OPTION]... PATTERN [FILE]... [10:46:14] Try '/bin/grep --help' for more information. [10:46:15] It's not mounted. [10:46:16] ;D [10:47:28] OH I GOT IT [10:51:32] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Puppet fails on integration instances: nfs_mount[home-on-labstoresvc]: umount: /home: not mounted - https://phabricator.wikimedia.org/T155820#2955655 (10hashar) On `integration-slave-jessie-1001` ``` # /usr/local/sbin/nfs-mount-manager... [10:51:50] yuvipanda: nfs-mount-manager has a grep that is too permissive [10:52:04] and when using 'check /home' it matches another mount: /srv/home/jenkins-deploy/tmpfs [10:56:04] yuvipanda: https://gerrit.wikimedia.org/r/333230 :-} [10:59:26] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Puppet fails on integration instances: nfs_mount[home-on-labstoresvc]: umount: /home: not mounted - https://phabricator.wikimedia.org/T155820#2955675 (10hashar) That matches the hosts failing puppet: ``` # salt --ou... [11:01:54] chasemp: fixed! have a good sleep :} [11:02:03] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Puppet fails on integration instances: nfs_mount[home-on-labstoresvc]: umount: /home: not mounted - https://phabricator.wikimedia.org/T155820#2955685 (10hashar) https://gerrit.wikimedia.org/r/#/c/333230/ cherry pick... [11:07:20] RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [11:10:49] RECOVERY - Puppet run on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [11:12:43] RECOVERY - Puppet run on integration-slave-precise-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [11:14:25] RECOVERY - Puppet run on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [11:14:45] RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [11:16:03] RECOVERY - Puppet run on integration-slave-precise-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [11:16:05] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [11:16:58] RECOVERY - Puppet run on integration-slave-precise-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [11:17:42] RECOVERY - Puppet run on integration-slave-trusty-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [11:18:52] RECOVERY - Puppet run on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [11:39:18] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Beta-Cluster-reproducible, 07Puppet: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929#2955738 (10hashar) That is still happening. Happened today when creat... [11:41:23] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Send out email regarding removal of jshint and jsonlint tests - https://phabricator.wikimedia.org/T155556#2946941 (10hashar) p:05Triage>03Normal [11:41:36] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 06Operations: Update npm to 3 or 4 - https://phabricator.wikimedia.org/T155488#2944663 (10hashar) p:05Triage>03Normal [11:42:17] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Use ccache for CI compilation? - https://phabricator.wikimedia.org/T152688#2955747 (10hashar) p:05Triage>03Low [11:42:52] PROBLEM - Host hashar-nfs is DOWN: CRITICAL - Host Unreachable (10.68.17.240) [11:45:41] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Send out email regarding removal of jshint and jsonlint tests - https://phabricator.wikimedia.org/T155556#2946941 (10Legoktm) There's absolutely no reason to remove jsonlint at this time. [11:49:21] 10Continuous-Integration-Config: Remove jshint from V+1 as well as V+2 pipes for repos that use eslint now - https://phabricator.wikimedia.org/T154924#2955759 (10hashar) [11:53:29] 10Continuous-Integration-Config, 13Patch-For-Review: Convert all repos to use npm with jsonlint and jshint|eslint - https://phabricator.wikimedia.org/T119973#2955768 (10hashar) [11:53:52] 10Continuous-Integration-Config, 13Patch-For-Review: Convert all repos to use npm with jsonlint and jshint|eslint - https://phabricator.wikimedia.org/T119973#1841667 (10hashar) I have repurposed this task to phase out the Jenkins jobs `jshint` and `jsonlint`. [11:54:07] 10Continuous-Integration-Config: Standalone jshint check for ULS is failing - https://phabricator.wikimedia.org/T144337#2955774 (10hashar) [11:54:09] 10Continuous-Integration-Config, 13Patch-For-Review: Convert all repos to use npm with jsonlint and jshint|eslint - https://phabricator.wikimedia.org/T119973#2955773 (10hashar) [11:54:31] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Send out email regarding removal of jshint and jsonlint tests - https://phabricator.wikimedia.org/T155556#2955776 (10hashar) [11:54:34] 10Continuous-Integration-Config, 13Patch-For-Review: Convert all repos to use npm with jsonlint and jshint|eslint - https://phabricator.wikimedia.org/T119973#1841667 (10hashar) [11:56:26] 10Browser-Tests-Infrastructure: Browser test Jenkins videos are corrupt - https://phabricator.wikimedia.org/T155794#2955777 (10zeljkofilipin) p:05Triage>03Normal [11:56:36] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Browser test Jenkins videos are corrupt - https://phabricator.wikimedia.org/T155794#2954941 (10zeljkofilipin) [11:58:15] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0] [11:58:53] 10Browser-Tests-Infrastructure, 10Wikidata, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#2955783 (10zeljkofilipin) [11:59:05] 10Browser-Tests-Infrastructure, 10Wikidata, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#2864860 (10zeljkofilipin) a:03zeljkofilipin [11:59:51] (03PS1) 10Hashar: [UniversalLanguageSelector] remove jshint/jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/333237 (https://phabricator.wikimedia.org/T144337) [12:05:21] (03CR) 10Hashar: [C: 032] [UniversalLanguageSelector] remove jshint/jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/333237 (https://phabricator.wikimedia.org/T144337) (owner: 10Hashar) [12:06:06] (03Merged) 10jenkins-bot: [UniversalLanguageSelector] remove jshint/jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/333237 (https://phabricator.wikimedia.org/T144337) (owner: 10Hashar) [12:06:32] 10Continuous-Integration-Config, 13Patch-For-Review: Standalone jshint check for ULS is failing - https://phabricator.wikimedia.org/T144337#2955816 (10hashar) 05Open>03Resolved a:03hashar [12:06:35] 10Continuous-Integration-Config, 13Patch-For-Review: Convert all repos to use npm with jsonlint and jshint|eslint - https://phabricator.wikimedia.org/T119973#2955818 (10hashar) [12:07:58] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [43200.0] [12:10:58] 10Continuous-Integration-Config, 13Patch-For-Review: Convert all repos to use npm Jenkins job with jsonlint and jshint|eslint - https://phabricator.wikimedia.org/T119973#2955820 (10hashar) [12:26:32] PROBLEM - Puppet staleness on deployment-aqs01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [43200.0] [13:16:22] 10Gerrit, 06Release-Engineering-Team, 10DBA, 06Operations, 13Patch-For-Review: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2955937 (10Paladox) I have never tested with a master and a replicate so not sure if it will work. I've only tested it on the main... [13:31:22] (03PS6) 10Hashar: Integration tests with make-wmf-branch [integration/config] - 10https://gerrit.wikimedia.org/r/256413 [13:31:32] RECOVERY - Puppet staleness on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [3600.0] [13:33:25] (03CR) 10jerkins-bot: [V: 04-1] Integration tests with make-wmf-branch [integration/config] - 10https://gerrit.wikimedia.org/r/256413 (owner: 10Hashar) [13:36:20] Yippee, build fixed! [13:36:20] Project selenium-Core » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #287: 09FIXED in 7 min 36 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/287/ [13:47:32] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #280: 04FAILURE in 2 min 32 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/280/ [13:52:35] (03PS7) 10Hashar: Integration tests with make-wmf-branch [integration/config] - 10https://gerrit.wikimedia.org/r/256413 [13:54:03] (03CR) 10jerkins-bot: [V: 04-1] Integration tests with make-wmf-branch [integration/config] - 10https://gerrit.wikimedia.org/r/256413 (owner: 10Hashar) [14:01:44] 10Continuous-Integration-Config, 10Gerrit, 10Wikidata: Archive wikidata/gremlin.git - https://phabricator.wikimedia.org/T155829#2956010 (10hashar) [14:02:51] 10Continuous-Integration-Config, 10Gerrit, 06Discovery, 10Wikidata, 10Wikidata-Query-Service: Archive wikidata/gremlin.git - https://phabricator.wikimedia.org/T155829#2956022 (10hashar) [14:22:40] (03PS1) 10Hashar: Wikibase: remove php53 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333251 [14:31:16] (03PS2) 10Hashar: [Wikibase]: remove php53 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333251 [14:31:18] (03PS1) 10Hashar: [Wikibase] remove dupe experimental jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333252 [14:32:19] (03PS1) 10Hashar: (WIP) Wikibase jobs on Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/333253 [14:35:30] (03CR) 10Thiemo Mättig (WMDE): [C: 031] "Wikibase is not even compatible with 5.3 any more. Was this even used?" [integration/config] - 10https://gerrit.wikimedia.org/r/333251 (owner: 10Hashar) [14:41:56] 10Gerrit, 06Release-Engineering-Team, 10DBA, 06Operations, 13Patch-For-Review: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2956039 (10Dzahn) >>! In T155764#2955567, @Marostegui wrote: > This is m2 and the other databases on this host are: > bugzilla_test... [14:42:29] (03PS1) 10Umherirrender: Add npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333256 [14:45:51] (03CR) 10Umherirrender: "The last one" [integration/config] - 10https://gerrit.wikimedia.org/r/333256 (owner: 10Umherirrender) [14:55:43] (03CR) 10Hashar: [C: 032] [Wikibase] remove dupe experimental jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333252 (owner: 10Hashar) [14:56:32] (03Merged) 10jenkins-bot: [Wikibase] remove dupe experimental jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333252 (owner: 10Hashar) [14:57:17] (03CR) 10Hashar: [C: 032] "Legacy cruft from when we still had to support 5.3 maybe... Thanks for the confirmation!" [integration/config] - 10https://gerrit.wikimedia.org/r/333251 (owner: 10Hashar) [14:58:44] (03Merged) 10jenkins-bot: [Wikibase]: remove php53 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333251 (owner: 10Hashar) [15:02:56] 10Continuous-Integration-Config, 05Goal, 07I18n, 13Patch-For-Review, 05WMF-deploy-2017-01-03_(1.29.0-wmf.7): Configure banana checker for i18n files to run on all MediaWiki extensions and skins - https://phabricator.wikimedia.org/T94547#2956053 (10Umherirrender) At the moment all extensions of mediawiki/... [15:13:37] 10Continuous-Integration-Config, 05Goal, 07I18n, 13Patch-For-Review, 05WMF-deploy-2017-01-03_(1.29.0-wmf.7): Configure banana checker for i18n files to run on all MediaWiki extensions and skins - https://phabricator.wikimedia.org/T94547#2956057 (10hashar) > At the moment all extensions of mediawiki/exten... [15:14:10] 10Continuous-Integration-Config, 05Goal, 07I18n, 13Patch-For-Review, 05WMF-deploy-2017-01-03_(1.29.0-wmf.7): Configure banana checker for i18n files to run on all MediaWiki extensions and skins - https://phabricator.wikimedia.org/T94547#2956058 (10hashar) [15:14:44] (03PS2) 10Hashar: Add npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333256 (owner: 10Umherirrender) [15:14:57] (03CR) 10Hashar: [C: 032] Add npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333256 (owner: 10Umherirrender) [15:15:59] (03Merged) 10jenkins-bot: Add npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/333256 (owner: 10Umherirrender) [15:29:06] (03Abandoned) 10Hashar: Add mw-run-phpunit-allskins.sh shell script for running phpunit in skins [integration/jenkins] - 10https://gerrit.wikimedia.org/r/291929 (owner: 10Paladox) [15:42:08] 10Gerrit, 06Release-Engineering-Team, 10DBA, 06Operations, 13Patch-For-Review: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2956092 (10Paladox) @demon would you think what @Marostegui said would work. (master to replication)? [16:02:22] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Browser test Jenkins videos are corrupt - https://phabricator.wikimedia.org/T155794#2954941 (10zeljkofilipin) @hashar did something change recently in Jenkins slaves that could cause the failure? [16:07:31] 10Continuous-Integration-Config, 07TestMe: fix or mark as inactive extensions currently failing CI - https://phabricator.wikimedia.org/T134090#2956149 (10Umherirrender) [16:11:36] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Send out email regarding removal of jshint and jsonlint tests - https://phabricator.wikimedia.org/T155556#2956164 (10Aklapper) > We should send out an email regarding removing jshint and jsonlint tests. The... [16:12:16] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Send out email regarding removal of jshint and jsonlint tests - https://phabricator.wikimedia.org/T155556#2956168 (10Aklapper) Ah, I guess that's related to T119973 ? I see. Sorry then :) [16:12:22] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Send out email regarding removal of jshint and jsonlint tests - https://phabricator.wikimedia.org/T155556#2956170 (10Paladox) >>! In T155556#2956164, @Aklapper wrote: >> We should send out an email regarding... [16:19:49] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Browser test Jenkins videos are corrupt - https://phabricator.wikimedia.org/T155794#2956178 (10zeljkofilipin) Looks like the video files are not corrupt. Steps to reproduce: - go to https://integration.wikimedia.org/ci/job/mwext-mw-selenium-jessie/725/ -... [16:26:44] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Browser test Jenkins videos are corrupt - https://phabricator.wikimedia.org/T155794#2956188 (10zeljkofilipin) Steps to reproduce the problem: - go to https://integration.wikimedia.org/ci/job/mwext-mw-selenium-jessie/725/ - **LEFT** click `Search: Search wi... [16:27:34] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Browser test Jenkins videos are corrupt - https://phabricator.wikimedia.org/T155794#2956189 (10zeljkofilipin) I am not sure what to do. [16:28:02] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Browser test Jenkins videos are corrupt - https://phabricator.wikimedia.org/T155794#2956190 (10zeljkofilipin) p:05Normal>03Low a:05zeljkofilipin>03None [16:35:57] (03PS1) 10Hashar: make-release: helper for branch -> version mapping [tools/release] - 10https://gerrit.wikimedia.org/r/333271 [16:43:28] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:48:00] (03CR) 10Chad: [C: 032] "This will be even easier once I get around to fixing T137564, then you won't need to figure it out yourself :)" [tools/release] - 10https://gerrit.wikimedia.org/r/333271 (owner: 10Hashar) [16:49:00] (03Merged) 10jenkins-bot: make-release: helper for branch -> version mapping [tools/release] - 10https://gerrit.wikimedia.org/r/333271 (owner: 10Hashar) [16:49:01] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Browser test Jenkins videos are corrupt - https://phabricator.wikimedia.org/T155794#2956224 (10Jdlrobson) Joaquin was able to download and view with vlc player. I used to be able to view these in browser. Did we change how we encoder build these videos ? [16:55:06] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Browser test Jenkins videos are corrupt - https://phabricator.wikimedia.org/T155794#2956232 (10hashar) The video is not corrupt for sure. On Chromium it downloads the video directly. On Safari it tries to plays it and fail. ``` $ curl --head 'https://integr... [17:00:10] ostriches: thx :) [17:00:59] yw [17:04:06] (03PS1) 10Hashar: Restore SecureHTML [integration/config] - 10https://gerrit.wikimedia.org/r/333274 (https://phabricator.wikimedia.org/T155153) [17:05:13] (03CR) 10Hashar: [C: 032] Restore SecureHTML [integration/config] - 10https://gerrit.wikimedia.org/r/333274 (https://phabricator.wikimedia.org/T155153) (owner: 10Hashar) [17:05:59] (03Merged) 10jenkins-bot: Restore SecureHTML [integration/config] - 10https://gerrit.wikimedia.org/r/333274 (https://phabricator.wikimedia.org/T155153) (owner: 10Hashar) [17:07:04] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression: doc.wikimedia.org displays "403 Forbidden" for coverage sub directories - https://phabricator.wikimedia.org/T150727#2956262 (10hashar) antoine-approve [17:15:06] 10Gerrit, 06Release-Engineering-Team, 10DBA, 06Operations, 13Patch-For-Review: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2956267 (10demon) >>! In T155764#2955567, @Marostegui wrote: > Given the size of the tables (very small ones) I would rather do the... [18:11:56] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Upgrade Jenkins from 1.x to latest 2.x - https://phabricator.wikimedia.org/T144106#2956392 (10Paladox) @hashar hi, has there been any progress on this? [18:13:34] (03PS1) 10Hashar: (WIP) mediawiki-releases job [integration/config] - 10https://gerrit.wikimedia.org/r/333280 [18:14:13] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Upgrade Jenkins to 1.651.3 - https://phabricator.wikimedia.org/T144105#2956395 (10Paladox) @hashar hi, are we going to do the 2.x update or are we going to do our last 1.x upgrade and then upgrade to 2.x as it looks like 1.x is n... [18:15:16] (03PS2) 10Hashar: (WIP) mediawiki-releases job [integration/config] - 10https://gerrit.wikimedia.org/r/333280 [18:19:49] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Install the blue ocean plugin alongside jenkins 2.x upgrade - https://phabricator.wikimedia.org/T155840#2956401 (10Paladox) [18:20:25] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Install the blue ocean plugin alongside jenkins 2.x upgrade - https://phabricator.wikimedia.org/T155840#2956417 (10Paladox) [18:20:28] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Upgrade Jenkins from 1.x to latest 2.x - https://phabricator.wikimedia.org/T144106#2956416 (10Paladox) [18:31:02] 10Gerrit, 10Developer-Wishlist (2017), 13Patch-For-Review: Allow Gerrit changes to be properly marked as WIP - https://phabricator.wikimedia.org/T155530#2956449 (10scfc) @hashar: Merci. I had forgotten about that. [18:31:25] 10Gerrit, 10Developer-Wishlist (2017), 07Documentation: Enable and document "WIP" workflow status in Gerrit - https://phabricator.wikimedia.org/T135245#2956450 (10scfc) [18:35:45] 10Continuous-Integration-Config, 10Gerrit, 06Discovery, 10Wikidata, 10Wikidata-Query-Service: Archive wikidata/gremlin.git - https://phabricator.wikimedia.org/T155829#2956454 (10Smalyshev) Yes, this is not developed anymore, can be archived. [18:37:47] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Beta-Cluster-reproducible, 07Puppet: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929#2736876 (10scfc) (T152941 is slightly related, but refers to the case... [19:23:44] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T154682#2956575 (10demon) 05Open>03Resolved [20:04:00] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 15User-greg: Facilitate Wikidev'17 main topic "How to manage our technical debt" - https://phabricator.wikimedia.org/T147937#2956631 (10greg) [20:37:19] 10Continuous-Integration-Config, 10BlueSpice, 07Easy, 07I18n: Extension BlueSpiceExtensions failing tests due to missing apihelp messages - https://phabricator.wikimedia.org/T155055#2956745 (10Phantom42) a:03Phantom42 [20:52:58] 06Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Weekly "misc MediaWiki extensions" deployment window proposal - https://phabricator.wikimedia.org/T148808#2956798 (10AndyRussG) Thanks so much for making this proposal!!!! Sounds great!! So, as far as I can tell, for de... [20:59:59] 10Gerrit, 07artificial-intelligence: Patch-wrangler -- suggests the best reviewers for a patch - https://phabricator.wikimedia.org/T155851#2956819 (10Halfak) [21:05:32] 06Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Weekly "misc MediaWiki extensions" deployment window proposal - https://phabricator.wikimedia.org/T148808#2733554 (10demon) I'm not a fan of this idea, as I think it further enshrines one of the special snowflakes (CN is... [21:39:17] 03Scap3: Package + deploy new version of git-fat - https://phabricator.wikimedia.org/T155856#2956925 (10demon) [21:40:01] 03Scap3, 06Operations: Package + deploy new version of git-fat - https://phabricator.wikimedia.org/T155856#2956963 (10demon) [21:41:44] PROBLEM - Puppet run on deployment-puppetdb01 is CRITICAL: CRITICAL: 16.67% of data above the critical threshold [0.0] [21:50:04] (03Abandoned) 10Zppix: Add functionality to email Wikidata mailing list upon build failure [integration/config] - 10https://gerrit.wikimedia.org/r/328315 (https://phabricator.wikimedia.org/T152495) (owner: 10Zppix) [21:52:25] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:55:09] 06Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Weekly "misc MediaWiki extensions" deployment window proposal - https://phabricator.wikimedia.org/T148808#2957009 (10greg) 05Open>03declined A bit of history... When I joined I (obviously) took ownership of the [[wi... [22:02:29] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [22:06:20] greg-g this may be a bit off topic, but I been wondering this for awhile, but you seem to be the head of a lot are you just the head of releng or mutiple teams? [22:13:12] 03Scap3, 06Operations: Package + deploy new version of git-fat - https://phabricator.wikimedia.org/T155856#2957067 (10thcipriani) `0.1.2-1` sounds reasonable to me. Changelog is reasonably thousandth-level-y Changelog --- * Add whitespace for readability * Some code pep8-ifying * Ensured that a file touch pre... [22:14:26] PROBLEM - Puppet run on deployment-puppetmaster02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:16:52] PROBLEM - Puppet run on deployment-fluorine02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:17:31] PROBLEM - Puppet run on deployment-zookeeper01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:18:31] PROBLEM - Puppet run on deployment-urldownloader is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:18:46] that'll be me [22:19:01] PROBLEM - Puppet run on deployment-mira is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:19:03] PROBLEM - Puppet run on deployment-copper is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:19:09] PROBLEM - Puppet run on deployment-jobrunner02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:19:13] PROBLEM - Puppet run on deployment-salt02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:19:15] PROBLEM - Puppet run on deployment-zotero01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:19:35] PROBLEM - Puppet run on deployment-elastic06 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:19:57] PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:20:23] PROBLEM - Puppet run on deployment-mcs01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:20:31] PROBLEM - Puppet run on deployment-pdfrender02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:20:49] PROBLEM - Puppet run on deployment-sca03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:20:58] PROBLEM - Puppet run on deployment-prometheus01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:21:24] Zppix: I am the Releaes Engineering Team Manager. Also "Release Manager" for WMF, so that ges me involved with a lot. [22:21:52] s/ges/gets/ [22:21:55] greg-g ah that makes sense. Are you a remote employee or do you go to physical location with in WMF? [22:22:08] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:22:22] PROBLEM - Puppet run on deployment-changeprop is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:22:34] I live ~40 miles north of SF, I only go in when I need to. I started out living in SF but was priced out. [22:23:02] wow, cool wish i lived in cali, but anyway it looks like deployment just died, is there an updating going on? [22:23:19] update rather. [22:23:22] 10Continuous-Integration-Config, 10VisualEditor: VisualEditor-MediaWiki: Don't be "smart" about only running jsduck on JS file changes - https://phabricator.wikimedia.org/T155862#2957097 (10Jdforrester-WMF) [22:23:57] 22:18 < Krenair> that'll be me [22:24:04] PROBLEM - Puppet run on deployment-apertium02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:24:06] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:24:20] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:24:47] greg-g ah ok, i missed that got all jumbled in with shinken-wm spam [22:25:04] PROBLEM - Puppet run on deployment-aqs01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:25:18] PROBLEM - Puppet run on deployment-trending01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:25:49] PROBLEM - Puppet run on deployment-mx is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:27:07] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:27:13] PROBLEM - Puppet run on deployment-mediawiki04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:27:31] PROBLEM - Puppet run on deployment-sca02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:28:27] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:28:31] PROBLEM - Puppet run on deployment-stream is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:29:17] PROBLEM - Puppet run on deployment-sca01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:29:17] PROBLEM - Puppet run on deployment-sentry01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:29:31] PROBLEM - Puppet run on deployment-poolcounter04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:29:35] PROBLEM - Puppet run on deployment-pdf01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:29:41] ok managed to revert [22:30:10] PROBLEM - Puppet run on deployment-ircd is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:30:14] PROBLEM - Puppet run on deployment-kafka03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:30:26] PROBLEM - Puppet run on deployment-secureredirexperiment is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:31:04] PROBLEM - Puppet run on deployment-ms-be01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:32:42] Krenair what were you attempting to do [22:33:19] puppetdb [22:35:27] performance-webpagetest-wmf doesnt test with IE or edge is there a reason for that? [22:36:03] Krenair sounds like fun, good thing its not me doing it. [22:36:11] indeed [22:39:25] RECOVERY - Puppet run on deployment-puppetmaster02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:41:51] question, whats the difference between shinken-wm and ichinga [22:42:08] icinga-wm* [22:43:28] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [22:43:55] icinga is the actual software [22:44:16] but they both are bots on irc shinken-wm though isnt used in -operations [22:44:17] used in prod [22:44:21] to replace nagios [22:44:35] that monitors pretty much everything over there [22:44:39] shinken is kind of used by some labs projects [22:46:48] Krenair but they are essentialy the same thing for the most part [22:49:56] They serve similar roles, but as usual, we have 2 solutions to one problem :D [22:50:48] ostriches thats the wmf way :P [22:52:28] is one in php and one in nodejs? [22:53:06] greg-g i wasnt aware php could run an irc bot. [22:54:49] it was a joke, and http://www.phpbots.org/ [22:55:04] any language can be made to speak a protocol [22:55:27] RECOVERY - Puppet run on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:55:31] RECOVERY - Puppet run on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:55:51] greg-g i perfer python atm its easier for me [22:56:53] RECOVERY - Puppet run on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:57:07] RECOVERY - Puppet run on deployment-phab02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:57:21] RECOVERY - Puppet run on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [22:57:29] RECOVERY - Puppet run on deployment-zookeeper01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:57:59] I once wrote some of the basics of an IRC server in PHP [22:58:29] RECOVERY - Puppet run on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [22:58:49] well i uh... uh created an index.php file xD [22:59:00] RECOVERY - Puppet run on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [22:59:03] RECOVERY - Puppet run on deployment-copper is OK: OK: Less than 1.00% above the threshold [0.0] [22:59:06] RECOVERY - Puppet run on deployment-apertium02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:59:10] RECOVERY - Puppet run on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:59:10] RECOVERY - Puppet run on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:59:14] RECOVERY - Puppet run on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:59:34] RECOVERY - Puppet run on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [22:59:56] RECOVERY - Puppet run on deployment-phab01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:00:46] RECOVERY - Puppet run on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0] [23:00:56] RECOVERY - Puppet run on deployment-prometheus01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:02:04] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [23:02:13] RECOVERY - Puppet run on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [23:03:31] RECOVERY - Puppet run on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [23:04:08] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [23:04:18] RECOVERY - Puppet run on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:04:20] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [23:05:08] RECOVERY - Puppet run on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:05:18] RECOVERY - Puppet run on deployment-trending01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:05:50] RECOVERY - Puppet run on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [23:07:30] RECOVERY - Puppet run on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:09:17] RECOVERY - Puppet run on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:09:33] RECOVERY - Puppet run on deployment-poolcounter04 is OK: OK: Less than 1.00% above the threshold [0.0] [23:09:35] RECOVERY - Puppet run on deployment-pdf01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:09:56] 10Beta-Cluster-Infrastructure, 10Flow, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017), 07Technical-Debt: Use dedicated $wgFlowCluster and $wgFlowDefaultWikiDb on Beta Cluster - https://phabricator.wikimedia.org/T147523#2957195 (10Mattflaschen-WMF) Should be discussed in conjunction with {T121827... [23:10:09] RECOVERY - Puppet run on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [23:10:15] RECOVERY - Puppet run on deployment-kafka03 is OK: OK: Less than 1.00% above the threshold [0.0] [23:10:25] RECOVERY - Puppet run on deployment-secureredirexperiment is OK: OK: Less than 1.00% above the threshold [0.0] [23:11:03] RECOVERY - Puppet run on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0]