[00:11:23] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [00:11:23] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [00:16:20] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [00:16:20] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [00:20:07] Project beta-update-databases-eqiad build #29935: 04STILL FAILING in 6.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29935/ [00:20:07] Project beta-update-databases-eqiad build #29935: 04STILL FAILING in 6.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29935/ [01:20:02] Project beta-update-databases-eqiad build #29936: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29936/ [01:20:02] Project beta-update-databases-eqiad build #29936: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29936/ [02:04:27] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) So while that was having the same old problems, puppet reversed all my swift changes. Didn't realise... [02:04:27] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) So while that was having the same old problems, puppet reversed all my swift changes. Didn't realise... [02:19:49] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) Hopefully Puppet won't do that again now. Running captcha generation again. [02:19:49] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) Hopefully Puppet won't do that again now. Running captcha generation again. [02:20:02] Project beta-update-databases-eqiad build #29937: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29937/ [02:20:02] Project beta-update-databases-eqiad build #29937: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29937/ [02:22:32] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) `krenair@deployment-deploy01:~$ sudo -u www-data php /srv/mediawiki/multiversion/MWScript.php extens... [02:22:32] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) `krenair@deployment-deploy01:~$ sudo -u www-data php /srv/mediawiki/multiversion/MWScript.php extens... [02:39:42] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) But when I go to test it I find the database is read-only for some reason. That doesn't make sense -... [02:39:42] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) But when I go to test it I find the database is read-only for some reason. That doesn't make sense -... [03:20:07] Project beta-update-databases-eqiad build #29938: 04STILL FAILING in 6.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29938/ [03:20:07] Project beta-update-databases-eqiad build #29938: 04STILL FAILING in 6.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29938/ [04:18:44] 10Project-Admins: Replace tracking bug T30486 by new project tag "Wikimedia-Incubator" - https://phabricator.wikimedia.org/T102506 (10Hydriz) @Aklapper Can we go ahead with this? It would be great to track Incubator site-specific tasks separately. [04:18:44] 10Project-Admins: Replace tracking bug T30486 by new project tag "Wikimedia-Incubator" - https://phabricator.wikimedia.org/T102506 (10Hydriz) @Aklapper Can we go ahead with this? It would be great to track Incubator site-specific tasks separately. [04:20:02] Project beta-update-databases-eqiad build #29939: 04STILL FAILING in 2.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29939/ [04:20:02] Project beta-update-databases-eqiad build #29939: 04STILL FAILING in 2.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29939/ [05:20:01] Project beta-update-databases-eqiad build #29940: 04STILL FAILING in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29940/ [05:20:01] Project beta-update-databases-eqiad build #29940: 04STILL FAILING in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29940/ [06:20:10] Project beta-update-databases-eqiad build #29941: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29941/ [06:20:10] Project beta-update-databases-eqiad build #29941: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29941/ [07:02:35] (03CR) 1020after4: [C: 032] deploy notes: Add train committers to blocker task [tools/release] - 10https://gerrit.wikimedia.org/r/471867 (owner: 10Thcipriani) [07:02:35] (03CR) 1020after4: [C: 032] deploy notes: Add train committers to blocker task [tools/release] - 10https://gerrit.wikimedia.org/r/471867 (owner: 10Thcipriani) [07:03:38] (03Merged) 10jenkins-bot: deploy notes: Add train committers to blocker task [tools/release] - 10https://gerrit.wikimedia.org/r/471867 (owner: 10Thcipriani) [07:03:39] (03Merged) 10jenkins-bot: deploy notes: Add train committers to blocker task [tools/release] - 10https://gerrit.wikimedia.org/r/471867 (owner: 10Thcipriani) [07:20:02] Project beta-update-databases-eqiad build #29942: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29942/ [07:20:02] Project beta-update-databases-eqiad build #29942: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29942/ [08:21:53] Project beta-update-databases-eqiad build #29943: 04STILL FAILING in 2.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29943/ [08:21:53] Project beta-update-databases-eqiad build #29943: 04STILL FAILING in 2.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29943/ [08:22:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733 (10fgiunchedi) >>! In T143733#4766671, @hashar wrote: > Jenkins emit the log to stdout or stder... [08:22:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733 (10fgiunchedi) >>! In T143733#4766671, @hashar wrote: > Jenkins emit the log to stdout or stder... [09:20:03] Project beta-update-databases-eqiad build #29944: 04STILL FAILING in 2.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29944/ [09:20:03] Project beta-update-databases-eqiad build #29944: 04STILL FAILING in 2.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29944/ [09:26:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733 (10hashar) >>! In T143733#4767255, @fgiunchedi wrote: > Agreed some context is lost via stdout/... [09:26:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733 (10hashar) >>! In T143733#4767255, @fgiunchedi wrote: > Agreed some context is lost via stdout/... [10:20:02] Project beta-update-databases-eqiad build #29945: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29945/ [10:20:02] Project beta-update-databases-eqiad build #29945: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29945/ [10:24:42] (03PS1) 10Hashar: docker: add Chrome deps to java8-wikidata-query-rdf [integration/config] - 10https://gerrit.wikimedia.org/r/475298 (https://phabricator.wikimedia.org/T209776) [10:24:42] (03PS1) 10Hashar: docker: add Chrome deps to java8-wikidata-query-rdf [integration/config] - 10https://gerrit.wikimedia.org/r/475298 (https://phabricator.wikimedia.org/T209776) [10:26:45] (03CR) 10Hashar: [C: 032] "That aligns releng/java8-wikidata-query-rdf which what is being used releng/npm-browser-tests." [integration/config] - 10https://gerrit.wikimedia.org/r/475298 (https://phabricator.wikimedia.org/T209776) (owner: 10Hashar) [10:26:45] (03CR) 10Hashar: [C: 032] "That aligns releng/java8-wikidata-query-rdf which what is being used releng/npm-browser-tests." [integration/config] - 10https://gerrit.wikimedia.org/r/475298 (https://phabricator.wikimedia.org/T209776) (owner: 10Hashar) [10:30:00] (03Merged) 10jenkins-bot: docker: add Chrome deps to java8-wikidata-query-rdf [integration/config] - 10https://gerrit.wikimedia.org/r/475298 (https://phabricator.wikimedia.org/T209776) (owner: 10Hashar) [10:30:00] (03Merged) 10jenkins-bot: docker: add Chrome deps to java8-wikidata-query-rdf [integration/config] - 10https://gerrit.wikimedia.org/r/475298 (https://phabricator.wikimedia.org/T209776) (owner: 10Hashar) [10:34:28] !log Building container releng/java8-wikidata-query-rdf:0.2.0 to support Chrome ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/475298/ | T209776 ) [10:34:28] !log Building container releng/java8-wikidata-query-rdf:0.2.0 to support Chrome ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/475298/ | T209776 ) [10:34:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:34:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:34:31] T209776: WDQS GUI build fails on CI - https://phabricator.wikimedia.org/T209776 [10:34:31] T209776: WDQS GUI build fails on CI - https://phabricator.wikimedia.org/T209776 [11:06:34] (03PS1) 10Hashar: Switch Jenkins job to releng/java8-wikidata-query-rdf:0.2.0 [integration/config] - 10https://gerrit.wikimedia.org/r/475305 (https://phabricator.wikimedia.org/T209776) [11:06:34] (03PS1) 10Hashar: Switch Jenkins job to releng/java8-wikidata-query-rdf:0.2.0 [integration/config] - 10https://gerrit.wikimedia.org/r/475305 (https://phabricator.wikimedia.org/T209776) [11:16:34] 10Continuous-Integration-Infrastructure (shipyard), 10Wikidata, 10Wikidata Query UI, 10Patch-For-Review: WDQS GUI build fails on CI - https://phabricator.wikimedia.org/T209776 (10hashar) I eventually though the issue got fixed but that was for the gui repository. For the rdf repository, I have added the `... [11:16:34] 10Continuous-Integration-Infrastructure (shipyard), 10Wikidata, 10Wikidata Query UI, 10Patch-For-Review: WDQS GUI build fails on CI - https://phabricator.wikimedia.org/T209776 (10hashar) I eventually though the issue got fixed but that was for the gui repository. For the rdf repository, I have added the `... [11:20:00] java8 issue bah [11:20:00] java8 issue bah [11:20:01] :( [11:20:01] :( [11:20:06] bbl [11:20:07] bbl [11:20:08] lunch & [11:20:08] lunch & [11:20:10] Project beta-update-databases-eqiad build #29946: 04STILL FAILING in 9.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29946/ [11:20:10] Project beta-update-databases-eqiad build #29946: 04STILL FAILING in 9.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29946/ [11:22:51] 10Beta-Cluster-Infrastructure, 10Analytics, 10User-Elukey: TCP connections between analytics and deployment-prep - https://phabricator.wikimedia.org/T208870 (10elukey) Thanks a lot for the link Tyler, I battled a bit with some settings (and updated the wiki as well) but eventually I was able to deploy on my... [11:22:51] 10Beta-Cluster-Infrastructure, 10Analytics, 10User-Elukey: TCP connections between analytics and deployment-prep - https://phabricator.wikimedia.org/T208870 (10elukey) Thanks a lot for the link Tyler, I battled a bit with some settings (and updated the wiki as well) but eventually I was able to deploy on my... [12:20:01] Project beta-update-databases-eqiad build #29947: 04STILL FAILING in 1.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29947/ [12:20:01] Project beta-update-databases-eqiad build #29947: 04STILL FAILING in 1.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29947/ [13:20:40] Project beta-update-databases-eqiad build #29948: 04STILL FAILING in 2.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29948/ [13:20:40] Project beta-update-databases-eqiad build #29948: 04STILL FAILING in 2.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29948/ [13:35:14] 10Scap, 10Operations: Canaries canaries canaries - https://phabricator.wikimedia.org/T210143 (10jijiki) p:05Triage>03Normal [13:35:14] 10Scap, 10Operations: Canaries canaries canaries - https://phabricator.wikimedia.org/T210143 (10jijiki) p:05Triage>03Normal [13:35:44] 10Scap, 10Operations, 10User-jijiki: Introduce state to Scap - https://phabricator.wikimedia.org/T209881 (10jijiki) [13:35:44] 10Scap, 10Operations, 10User-jijiki: Introduce state to Scap - https://phabricator.wikimedia.org/T209881 (10jijiki) [13:35:47] 10Scap, 10Operations: Canaries canaries canaries - https://phabricator.wikimedia.org/T210143 (10jijiki) [13:35:47] 10Scap, 10Operations: Canaries canaries canaries - https://phabricator.wikimedia.org/T210143 (10jijiki) [13:37:57] 10Project-Admins: Replace tracking bug T30486 by new project tag "Wikimedia-Incubator" - https://phabricator.wikimedia.org/T102506 (10Aklapper) Sure, feel free to go ahead. See https://www.mediawiki.org/wiki/Phabricator/Project_management/Tracking_tasks I'd prefer to call this project `incubator.wikimedia.org`... [13:37:57] 10Project-Admins: Replace tracking bug T30486 by new project tag "Wikimedia-Incubator" - https://phabricator.wikimedia.org/T102506 (10Aklapper) Sure, feel free to go ahead. See https://www.mediawiki.org/wiki/Phabricator/Project_management/Tracking_tasks I'd prefer to call this project `incubator.wikimedia.org`... [14:14:53] 10Continuous-Integration-Infrastructure (shipyard), 10Wikidata, 10Wikidata Query UI, 10Patch-For-Review: WDQS GUI build fails on CI - https://phabricator.wikimedia.org/T209776 (10hashar) The job now crashes with: [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM term... [14:14:53] 10Continuous-Integration-Infrastructure (shipyard), 10Wikidata, 10Wikidata Query UI, 10Patch-For-Review: WDQS GUI build fails on CI - https://phabricator.wikimedia.org/T209776 (10hashar) The job now crashes with: [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM term... [14:20:02] Project beta-update-databases-eqiad build #29949: 04STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29949/ [14:20:02] Project beta-update-databases-eqiad build #29949: 04STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29949/ [14:51:46] 10Project-Admins: Create project: TWLCon-2019India - https://phabricator.wikimedia.org/T210150 (10Krishna_Chaitanya_Velaga) [14:51:46] 10Project-Admins: Create project: TWLCon-2019India - https://phabricator.wikimedia.org/T210150 (10Krishna_Chaitanya_Velaga) [14:53:46] 10Project-Admins: Create project: TWLCon-2019India - https://phabricator.wikimedia.org/T210150 (10Aklapper) Hi, is there any page which provides more information about TWLCon, to which the Phab project description could link? I guess for TWL we could link to https://meta.wikimedia.org/wiki/The_Wikipedia_Library ? [14:53:46] 10Project-Admins: Create project: TWLCon-2019India - https://phabricator.wikimedia.org/T210150 (10Aklapper) Hi, is there any page which provides more information about TWLCon, to which the Phab project description could link? I guess for TWL we could link to https://meta.wikimedia.org/wiki/The_Wikipedia_Library ? [15:20:11] Project beta-update-databases-eqiad build #29950: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29950/ [15:20:11] Project beta-update-databases-eqiad build #29950: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29950/ [15:21:40] 10Project-Admins: Create project: TWLCon-2019India - https://phabricator.wikimedia.org/T210150 (10Krishna_Chaitanya_Velaga) @Aklapper - Yes, TWL can be linked to https://meta.wikimedia.org/wiki/The_Wikipedia_Library - The event page is at https://meta.wikimedia.org/wiki/TWLCon_(2019_India) [15:21:41] 10Project-Admins: Create project: TWLCon-2019India - https://phabricator.wikimedia.org/T210150 (10Krishna_Chaitanya_Velaga) @Aklapper - Yes, TWL can be linked to https://meta.wikimedia.org/wiki/The_Wikipedia_Library - The event page is at https://meta.wikimedia.org/wiki/TWLCon_(2019_India) [15:27:20] (03CR) 10Hashar: [C: 032] "Deployed, and it works on https://gerrit.wikimedia.org/r/#/c/wikidata/query/rdf/+/474391/" [integration/config] - 10https://gerrit.wikimedia.org/r/475305 (https://phabricator.wikimedia.org/T209776) (owner: 10Hashar) [15:27:20] (03CR) 10Hashar: [C: 032] "Deployed, and it works on https://gerrit.wikimedia.org/r/#/c/wikidata/query/rdf/+/474391/" [integration/config] - 10https://gerrit.wikimedia.org/r/475305 (https://phabricator.wikimedia.org/T209776) (owner: 10Hashar) [15:32:26] (03Merged) 10jenkins-bot: Switch Jenkins job to releng/java8-wikidata-query-rdf:0.2.0 [integration/config] - 10https://gerrit.wikimedia.org/r/475305 (https://phabricator.wikimedia.org/T209776) (owner: 10Hashar) [15:32:26] (03Merged) 10jenkins-bot: Switch Jenkins job to releng/java8-wikidata-query-rdf:0.2.0 [integration/config] - 10https://gerrit.wikimedia.org/r/475305 (https://phabricator.wikimedia.org/T209776) (owner: 10Hashar) [15:52:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733 (10hashar) `counterexample /src/gui/node_modules/puppeteer/.local-chromium/linux-579032/chrome-... [15:52:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733 (10hashar) `counterexample /src/gui/node_modules/puppeteer/.local-chromium/linux-579032/chrome-... [15:56:53] 10Project-Admins: Replace tracking bug T30486 by new project tag "Wikimedia-Incubator" - https://phabricator.wikimedia.org/T102506 (10Hydriz) @Aklapper Any place for me to request for batch-editing rights? [15:56:53] 10Project-Admins: Replace tracking bug T30486 by new project tag "Wikimedia-Incubator" - https://phabricator.wikimedia.org/T102506 (10Hydriz) @Aklapper Any place for me to request for batch-editing rights? [16:20:02] Project beta-update-databases-eqiad build #29951: 04STILL FAILING in 2.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29951/ [16:20:02] Project beta-update-databases-eqiad build #29951: 04STILL FAILING in 2.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29951/ [16:52:32] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Kubernetes: set up a test node with new version, Redis as cache, a new Swift container and export metrics over Fraphana - https://phabricator.wikimedia.org/T210076 (10fselles) i've got some metrics at least. ` # HELP go_gc_duration_seconds... [16:52:32] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Kubernetes: set up a test node with new version, Redis as cache, a new Swift container and export metrics over Fraphana - https://phabricator.wikimedia.org/T210076 (10fselles) i've got some metrics at least. ` # HELP go_gc_duration_seconds... [17:20:06] Project beta-update-databases-eqiad build #29952: 04STILL FAILING in 6.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29952/ [17:20:07] Project beta-update-databases-eqiad build #29952: 04STILL FAILING in 6.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29952/ [18:07:50] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10PerfektesChaos) [18:07:50] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10PerfektesChaos) [18:07:52] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10PerfektesChaos) 05Resolved>03Open Reopening: Unfortunately not yet fully recovered. enwiki@BETA is fine for me. [[ https://de.wikipedia.beta.wmf... [18:07:52] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10PerfektesChaos) 05Resolved>03Open Reopening: Unfortunately not yet fully recovered. enwiki@BETA is fine for me. [[ https://de.wikipedia.beta.wmf... [18:20:02] Project beta-update-databases-eqiad build #29953: 04STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29953/ [18:20:02] Project beta-update-databases-eqiad build #29953: 04STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29953/ [19:20:02] Project beta-update-databases-eqiad build #29954: 04STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29954/ [19:20:02] Project beta-update-databases-eqiad build #29954: 04STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29954/ [20:12:58] so deployment-prep [20:12:58] so deployment-prep [20:13:02] DBs are RO, VE is out of action (HTTP 503s) [20:13:02] DBs are RO, VE is out of action (HTTP 503s) [20:16:38] oh and I need to update the MW varnish purging config [20:16:38] oh and I need to update the MW varnish purging config [20:17:40] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) [20:17:40] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) [20:17:43] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10Krenair) 05Open>03Resolved That does not seem to be this task @PerfektesChaos [20:17:43] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10Krenair) 05Open>03Resolved That does not seem to be this task @PerfektesChaos [20:19:41] https://gerrit.wikimedia.org/r/475363 [20:19:42] https://gerrit.wikimedia.org/r/475363 [20:20:07] Project beta-update-databases-eqiad build #29955: 04STILL FAILING in 6.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29955/ [20:20:07] Project beta-update-databases-eqiad build #29955: 04STILL FAILING in 6.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29955/ [20:23:19] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) Also: * DBs are RO * VE is out of action (HTTP 503s) [20:23:19] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) Also: * DBs are RO * VE is out of action (HTTP 503s) [20:38:49] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint): Flaky unit test Flow\Tests\Collection\PostCollectionTest::testGetLastRevision - https://phabricator.wikimedia.org/T210204 (10D3r1ck01) p:05Triage>03Normal [20:38:49] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint): Flaky unit test Flow\Tests\Collection\PostCollectionTest::testGetLastRevision - https://phabricator.wikimedia.org/T210204 (10D3r1ck01) p:05Triage>03Normal [20:44:39] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint), and 2 others: Flaky unit test Flow\Tests\Collection\PostCollectionTest::testGetLastRevision - https://phabricator.wikimedia.org/T210204 (10D3r1ck01) a:03D3r1ck01 [20:44:39] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint), and 2 others: Flaky unit test Flow\Tests\Collection\PostCollectionTest::testGetLastRevision - https://phabricator.wikimedia.org/T210204 (10D3r1ck01) a:03D3r1ck01 [20:56:30] why do our SSH sessions into labs break so easily after inactivity [20:56:30] why do our SSH sessions into labs break so easily after inactivity [20:58:36] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) 2018-11-22 20:48:10 [W-cWCawQBHcAAErbfF0AAAAH] deployment-mediawiki-07 enwiki 1.33.0-alpha DBReplicat... [20:58:36] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) 2018-11-22 20:48:10 [W-cWCawQBHcAAErbfF0AAAAH] deployment-mediawiki-07 enwiki 1.33.0-alpha DBReplicat... [21:20:02] Project beta-update-databases-eqiad build #29956: 04STILL FAILING in 2.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29956/ [21:20:02] Project beta-update-databases-eqiad build #29956: 04STILL FAILING in 2.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29956/ [21:23:21] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Wikidata, 10Wikidata Query UI, 10Patch-For-Review: WDQS GUI build fails on CI - https://phabricator.wikimedia.org/T209776 (10hashar) `counterexample /src/gui/node_modules/puppeteer/.local-chromium/linux-579032/chro... [21:23:21] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Wikidata, 10Wikidata Query UI, 10Patch-For-Review: WDQS GUI build fails on CI - https://phabricator.wikimedia.org/T209776 (10hashar) `counterexample /src/gui/node_modules/puppeteer/.local-chromium/linux-579032/chro... [21:24:12] Project beta-scap-eqiad build #228502: 04FAILURE in 4 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/228502/ [21:24:12] Project beta-scap-eqiad build #228502: 04FAILURE in 4 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/228502/ [21:24:55] ^ that was me [21:24:55] ^ that was me [21:27:21] Project beta-scap-eqiad build #228503: 04STILL FAILING in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/228503/ [21:27:22] Project beta-scap-eqiad build #228503: 04STILL FAILING in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/228503/ [21:40:00] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) `root@BETA[(none)]> show slave status \G *************************** 1. row ************************... [21:40:00] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) `root@BETA[(none)]> show slave status \G *************************** 1. row ************************... [21:48:54] Yippee, build fixed! [21:48:54] Yippee, build fixed! [21:48:54] Project beta-scap-eqiad build #228504: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/228504/ [21:48:54] Project beta-scap-eqiad build #228504: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/228504/ [22:02:06] (03PS1) 10Hashar: docker: puppeteer tweaks for java8-wikidata-query-rdf [integration/config] - 10https://gerrit.wikimedia.org/r/475428 (https://phabricator.wikimedia.org/T209776) [22:02:06] (03PS1) 10Hashar: docker: puppeteer tweaks for java8-wikidata-query-rdf [integration/config] - 10https://gerrit.wikimedia.org/r/475428 (https://phabricator.wikimedia.org/T209776) [22:02:08] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) Turns out that Slave_SQL_Running: No is a Bad Thing - despite it saying its waiting for master to sen... [22:02:08] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) Turns out that Slave_SQL_Running: No is a Bad Thing - despite it saying its waiting for master to sen... [22:07:19] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Wikidata, 10Wikidata Query UI, 10Patch-For-Review: WDQS GUI build fails on CI - https://phabricator.wikimedia.org/T209776 (10hashar) **Summary** https://gerrit.wikimedia.org/r/#/c/wikidata/query/gui/+/475427/ woul... [22:07:19] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Wikidata, 10Wikidata Query UI, 10Patch-For-Review: WDQS GUI build fails on CI - https://phabricator.wikimedia.org/T209776 (10hashar) **Summary** https://gerrit.wikimedia.org/r/#/c/wikidata/query/gui/+/475427/ woul... [22:12:16] 10Release-Engineering-Team (Backlog), 10Operations: Keyholder phab repo duplicate work - https://phabricator.wikimedia.org/T203003 (10hashar) I guess we can close rKEYHOLDER. Seems to me keyholder code will be moved out of operations/puppet to `operations/software/keyholder` where development has been occurrin... [22:12:16] 10Release-Engineering-Team (Backlog), 10Operations: Keyholder phab repo duplicate work - https://phabricator.wikimedia.org/T203003 (10hashar) I guess we can close rKEYHOLDER. Seems to me keyholder code will be moved out of operations/puppet to `operations/software/keyholder` where development has been occurrin... [22:14:32] 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 (10hashar) Eventually I felt migrating castor03 just before ThanksGiving to be a terrible idea. Postponed to next week. [22:14:32] 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 (10hashar) Eventually I felt migrating castor03 just before ThanksGiving to be a terrible idea. Postponed to next week. [22:20:23] Project beta-update-databases-eqiad build #29957: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29957/ [22:20:23] Project beta-update-databases-eqiad build #29957: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29957/ [22:32:50] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) Alright so I made it skip insertion of duplicate things until replication recovered and Seconds_Behin... [22:32:50] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) Alright so I made it skip insertion of duplicate things until replication recovered and Seconds_Behin... [22:59:49] (03CR) 10Pmiazga: [C: 04-1] "please refer to my comment in Ib94d7bbdf1a3edf6f8c0fecfc4c69b3625109a96" [integration/config] - 10https://gerrit.wikimedia.org/r/475428 (https://phabricator.wikimedia.org/T209776) (owner: 10Hashar) [22:59:49] (03CR) 10Pmiazga: [C: 04-1] "please refer to my comment in Ib94d7bbdf1a3edf6f8c0fecfc4c69b3625109a96" [integration/config] - 10https://gerrit.wikimedia.org/r/475428 (https://phabricator.wikimedia.org/T209776) (owner: 10Hashar) [23:04:16] (03CR) 10Pmiazga: [C: 031] docker: puppeteer tweaks for java8-wikidata-query-rdf (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/475428 (https://phabricator.wikimedia.org/T209776) (owner: 10Hashar) [23:04:16] (03CR) 10Pmiazga: [C: 031] docker: puppeteer tweaks for java8-wikidata-query-rdf (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/475428 (https://phabricator.wikimedia.org/T209776) (owner: 10Hashar) [23:20:12] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) As for VE: `root@deployment-cache-text05:/etc/varnish# grep deployment-restbase * -A1 wikimedia-commo... [23:20:12] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) As for VE: `root@deployment-cache-text05:/etc/varnish# grep deployment-restbase * -A1 wikimedia-commo... [23:21:24] Project beta-update-databases-eqiad build #29958: 04STILL FAILING in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29958/ [23:21:24] Project beta-update-databases-eqiad build #29958: 04STILL FAILING in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29958/