[00:20:26] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [00:20:26] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [00:25:22] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [00:25:22] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [01:20:11] Project beta-update-databases-eqiad build #30008: 04FAILURE in 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30008/ [01:20:11] Project beta-update-databases-eqiad build #30008: 04FAILURE in 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30008/ [01:42:22] Hm. unable to view https://phabricator.wikimedia.org/T204757 for some reason. Somethig about an epoch failure [01:42:22] Hm. unable to view https://phabricator.wikimedia.org/T204757 for some reason. Somethig about an epoch failure [01:42:49] Must be halfak__'s fault :P [01:42:49] Must be halfak__'s fault :P [01:50:52] Urgh. Same here. Please file a bug. [01:50:52] Urgh. Same here. Please file a bug. [01:51:25] and mention https://secure.phabricator.com/T9465 [01:51:25] and mention https://secure.phabricator.com/T9465 [01:51:29] Krinkle, ^ [01:51:29] Krinkle, ^ [01:51:59] andre__: Ah you mean, open a discussn thread on the phab communimty? [01:51:59] andre__: Ah you mean, open a discussn thread on the phab communimty? [01:52:19] I'd file a ticket in our instance, to start with [01:52:19] I'd file a ticket in our instance, to start with [01:54:25] 10Phabricator (Upstream), 10Upstream: Phabricator Exception: Construction of a DateTime() with epoch raised - https://phabricator.wikimedia.org/T210338 (10Krinkle) [01:54:25] 10Phabricator (Upstream), 10Upstream: Phabricator Exception: Construction of a DateTime() with epoch raised - https://phabricator.wikimedia.org/T210338 (10Krinkle) [01:54:56] 10Phabricator (Upstream), 10Upstream: Phabricator Exception: Construction of a DateTime() with epoch raised - https://phabricator.wikimedia.org/T210338 (10Krinkle) [01:54:56] 10Phabricator (Upstream), 10Upstream: Phabricator Exception: Construction of a DateTime() with epoch raised - https://phabricator.wikimedia.org/T210338 (10Krinkle) [01:58:35] 10Phabricator (Upstream), 10Upstream: Phabricator Exception: Construction of a DateTime() with epoch raised - https://phabricator.wikimedia.org/T210338 (10Aklapper) @mmodell (or someone else to ping here?), could you look up the corresponding stacktrace? [01:58:35] 10Phabricator (Upstream), 10Upstream: Phabricator Exception: Construction of a DateTime() with epoch raised - https://phabricator.wikimedia.org/T210338 (10Aklapper) @mmodell (or someone else to ping here?), could you look up the corresponding stacktrace? [02:20:02] Project beta-update-databases-eqiad build #30009: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30009/ [02:20:02] Project beta-update-databases-eqiad build #30009: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30009/ [03:10:49] Krinkle, if it helps I have a load of mail from that task [03:10:49] Krinkle, if it helps I have a load of mail from that task [03:20:06] Project beta-update-databases-eqiad build #30010: 04STILL FAILING in 6.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30010/ [03:20:06] Project beta-update-databases-eqiad build #30010: 04STILL FAILING in 6.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30010/ [03:34:37] 10Beta-Cluster-Infrastructure: beta-update-databases-eqiad is failing with a composer problem - https://phabricator.wikimedia.org/T210341 (10Krenair) [03:34:37] 10Beta-Cluster-Infrastructure: beta-update-databases-eqiad is failing with a composer problem - https://phabricator.wikimedia.org/T210341 (10Krenair) [03:35:30] 10Beta-Cluster-Infrastructure, 10Cloud-VPS: Puppet errors on various deployment-prep hosts - https://phabricator.wikimedia.org/T208879 (10Krenair) 05Open>03Resolved thanks all [03:35:30] 10Beta-Cluster-Infrastructure, 10Cloud-VPS: Puppet errors on various deployment-prep hosts - https://phabricator.wikimedia.org/T208879 (10Krenair) 05Open>03Resolved thanks all [03:38:26] 10Beta-Cluster-Infrastructure, 10Cloud-VPS: deployment-prep: Hosts haven't applied new firewall rules - https://phabricator.wikimedia.org/T209466 (10Krenair) 05Open>03Resolved probably partially taken care of in T208879, according to cumin everything (with ferm) has MONITORING_HOSTS 172.16.7.178 note this... [03:38:26] 10Beta-Cluster-Infrastructure, 10Cloud-VPS: deployment-prep: Hosts haven't applied new firewall rules - https://phabricator.wikimedia.org/T209466 (10Krenair) 05Open>03Resolved probably partially taken care of in T208879, according to cumin everything (with ferm) has MONITORING_HOSTS 172.16.7.178 note this... [03:52:54] 10Release-Engineering-Team, 10Language-strategy, 10MediaWiki-extensions-WikimediaIncubator, 10incubator.wikimedia.org, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10Hydriz) [03:52:54] 10Release-Engineering-Team, 10Language-strategy, 10MediaWiki-extensions-WikimediaIncubator, 10incubator.wikimedia.org, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10Hydriz) [03:53:36] 10Deployments, 10incubator.wikimedia.org: Enable LocalisationUpdate extension on Incubator - https://phabricator.wikimedia.org/T21312 (10Hydriz) [03:53:36] 10Deployments, 10incubator.wikimedia.org: Enable LocalisationUpdate extension on Incubator - https://phabricator.wikimedia.org/T21312 (10Hydriz) [03:59:27] 10Project-Admins, 10Epic: Replace all "tracking" tasks with tags/projects (if unbounded), relabel them to outcome tasks (if bounded), or kill them entirely (if pointless) - https://phabricator.wikimedia.org/T192655 (10Hydriz) [03:59:27] 10Project-Admins, 10Epic: Replace all "tracking" tasks with tags/projects (if unbounded), relabel them to outcome tasks (if bounded), or kill them entirely (if pointless) - https://phabricator.wikimedia.org/T192655 (10Hydriz) [03:59:29] 10Project-Admins: Replace tracking bug T30486 by new project tag "Wikimedia-Incubator" - https://phabricator.wikimedia.org/T102506 (10Hydriz) 05Open>03Resolved #incubator.wikimedia.org was created to replace T30486. [03:59:29] 10Project-Admins: Replace tracking bug T30486 by new project tag "Wikimedia-Incubator" - https://phabricator.wikimedia.org/T102506 (10Hydriz) 05Open>03Resolved #incubator.wikimedia.org was created to replace T30486. [04:03:28] 10Release-Engineering-Team, 10Language-strategy, 10incubator.wikimedia.org, 10Epic, 10I18n: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10Hydriz) [04:03:28] 10Release-Engineering-Team, 10Language-strategy, 10incubator.wikimedia.org, 10Epic, 10I18n: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10Hydriz) [04:20:01] Project beta-update-databases-eqiad build #30011: 04STILL FAILING in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30011/ [04:20:02] Project beta-update-databases-eqiad build #30011: 04STILL FAILING in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30011/ [05:06:22] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [05:06:22] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [05:16:22] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [05:16:22] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [05:19:39] 10Release-Engineering-Team, 10Language-strategy, 10MediaWiki-extensions-WikimediaIncubator, 10incubator.wikimedia.org, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10Amire80) [05:19:39] 10Release-Engineering-Team, 10Language-strategy, 10MediaWiki-extensions-WikimediaIncubator, 10incubator.wikimedia.org, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10Amire80) [05:20:10] Project beta-update-databases-eqiad build #30012: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30012/ [05:20:10] Project beta-update-databases-eqiad build #30012: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30012/ [06:20:01] Project beta-update-databases-eqiad build #30013: 04STILL FAILING in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30013/ [06:20:01] Project beta-update-databases-eqiad build #30013: 04STILL FAILING in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30013/ [07:05:08] 10Release-Engineering-Team, 10Language-strategy, 10MediaWiki-extensions-WikimediaIncubator, 10incubator.wikimedia.org, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10Hydriz) @Amire80 Anything that the extension can do in this task? It seems to me th... [07:05:08] 10Release-Engineering-Team, 10Language-strategy, 10MediaWiki-extensions-WikimediaIncubator, 10incubator.wikimedia.org, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10Hydriz) @Amire80 Anything that the extension can do in this task? It seems to me th... [07:20:05] Project beta-update-databases-eqiad build #30014: 04STILL FAILING in 5.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30014/ [07:20:06] Project beta-update-databases-eqiad build #30014: 04STILL FAILING in 5.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30014/ [08:20:02] Project beta-update-databases-eqiad build #30015: 04STILL FAILING in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30015/ [08:20:02] Project beta-update-databases-eqiad build #30015: 04STILL FAILING in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30015/ [08:22:11] 10Release-Engineering-Team, 10Language-strategy, 10MediaWiki-extensions-WikimediaIncubator, 10incubator.wikimedia.org, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10Amire80) Nothing immediate, but it's close enough to the topic. This project is ne... [08:22:11] 10Release-Engineering-Team, 10Language-strategy, 10MediaWiki-extensions-WikimediaIncubator, 10incubator.wikimedia.org, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10Amire80) Nothing immediate, but it's close enough to the topic. This project is ne... [09:20:11] Project beta-update-databases-eqiad build #30016: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30016/ [09:20:11] Project beta-update-databases-eqiad build #30016: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30016/ [10:17:24] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [10:17:24] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [10:20:01] Project beta-update-databases-eqiad build #30017: 04STILL FAILING in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30017/ [10:20:02] Project beta-update-databases-eqiad build #30017: 04STILL FAILING in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30017/ [10:27:24] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [10:27:24] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [11:20:01] Project beta-update-databases-eqiad build #30018: 04STILL FAILING in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30018/ [11:20:01] Project beta-update-databases-eqiad build #30018: 04STILL FAILING in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30018/ [12:20:06] Project beta-update-databases-eqiad build #30019: 04STILL FAILING in 6.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30019/ [12:20:06] Project beta-update-databases-eqiad build #30019: 04STILL FAILING in 6.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30019/ [12:35:38] 10Phabricator (Upstream), 10Upstream: Phabricator Exception: Construction of a DateTime() with epoch raised - https://phabricator.wikimedia.org/T210338 (10Aklapper) One of the last changes on that ticket (according to my inbox) was ` Krinkle removed Release Version. Krinkle removed Release Date. Herald changed... [12:35:38] 10Phabricator (Upstream), 10Upstream: Phabricator Exception: Construction of a DateTime() with epoch raised - https://phabricator.wikimedia.org/T210338 (10Aklapper) One of the last changes on that ticket (according to my inbox) was ` Krinkle removed Release Version. Krinkle removed Release Date. Herald changed... [13:20:06] Project beta-update-databases-eqiad build #30020: 04STILL FAILING in 5.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30020/ [13:20:06] Project beta-update-databases-eqiad build #30020: 04STILL FAILING in 5.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30020/ [14:20:01] Project beta-update-databases-eqiad build #30021: 04STILL FAILING in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30021/ [14:20:01] Project beta-update-databases-eqiad build #30021: 04STILL FAILING in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30021/ [15:20:06] Project beta-update-databases-eqiad build #30022: 04STILL FAILING in 5.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30022/ [15:20:06] Project beta-update-databases-eqiad build #30022: 04STILL FAILING in 5.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30022/ [15:22:01] PROBLEM - Host integration-slave-jessie-android is DOWN: CRITICAL - Host Unreachable (10.68.19.239) [15:22:02] PROBLEM - Host integration-slave-jessie-android is DOWN: CRITICAL - Host Unreachable (10.68.19.239) [15:26:32] PROBLEM - Host integration-slave-docker-1043 is DOWN: CRITICAL - Host Unreachable (10.68.22.120) [15:26:32] PROBLEM - Host integration-slave-docker-1043 is DOWN: CRITICAL - Host Unreachable (10.68.22.120) [16:20:01] Project beta-update-databases-eqiad build #30023: 04STILL FAILING in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30023/ [16:20:01] Project beta-update-databases-eqiad build #30023: 04STILL FAILING in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30023/ [17:20:11] Project beta-update-databases-eqiad build #30024: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30024/ [17:20:11] Project beta-update-databases-eqiad build #30024: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30024/ [18:17:25] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [18:17:25] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [18:20:01] Project beta-update-databases-eqiad build #30025: 04STILL FAILING in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30025/ [18:20:01] Project beta-update-databases-eqiad build #30025: 04STILL FAILING in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30025/ [18:27:22] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [18:27:22] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [19:20:06] Project beta-update-databases-eqiad build #30026: 04STILL FAILING in 5.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30026/ [19:20:06] Project beta-update-databases-eqiad build #30026: 04STILL FAILING in 5.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30026/ [20:20:01] Project beta-update-databases-eqiad build #30027: 04STILL FAILING in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30027/ [20:20:01] Project beta-update-databases-eqiad build #30027: 04STILL FAILING in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30027/ [20:38:53] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/urbanecmbot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/475619 [20:38:53] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/urbanecmbot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/475619 [20:38:56] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [labs/tools/urbanecmbot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/475619 (owner: 10QChris) [20:38:56] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [labs/tools/urbanecmbot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/475619 (owner: 10QChris) [20:39:16] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/urbanecmbot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/475620 [20:39:16] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/urbanecmbot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/475620 [20:39:18] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [labs/tools/urbanecmbot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/475620 (owner: 10QChris) [20:39:18] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [labs/tools/urbanecmbot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/475620 (owner: 10QChris) [21:20:06] Project beta-update-databases-eqiad build #30028: 04STILL FAILING in 6.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30028/ [21:20:06] Project beta-update-databases-eqiad build #30028: 04STILL FAILING in 6.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30028/ [22:20:06] Project beta-update-databases-eqiad build #30029: 04STILL FAILING in 6.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30029/ [22:20:06] Project beta-update-databases-eqiad build #30029: 04STILL FAILING in 6.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30029/ [23:20:01] Project beta-update-databases-eqiad build #30030: 04STILL FAILING in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30030/ [23:20:02] Project beta-update-databases-eqiad build #30030: 04STILL FAILING in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30030/