[00:10:45] 10Continuous-Integration-Infrastructure: Depends-On should not work across branches - https://phabricator.wikimedia.org/T210685 (10Catrope) [00:10:46] 10Continuous-Integration-Infrastructure: Depends-On should not work across branches - https://phabricator.wikimedia.org/T210685 (10Catrope) [00:21:23] Project beta-update-databases-eqiad build #30103: 04FAILURE in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30103/ [00:21:23] Project beta-update-databases-eqiad build #30103: 04FAILURE in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30103/ [00:59:06] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: Scribunto test “LuaStandalone: SecurityTests[1]: CVE-2014-5461” failing in Wikibase CI builds - https://phabricator.wikimedia.org/T210634 (10matmarex) Also occurred on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/4... [00:59:06] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: Scribunto test “LuaStandalone: SecurityTests[1]: CVE-2014-5461” failing in Wikibase CI builds - https://phabricator.wikimedia.org/T210634 (10matmarex) Also occurred on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/4... [01:21:22] Yippee, build fixed! [01:21:22] Yippee, build fixed! [01:21:23] Project beta-update-databases-eqiad build #30104: 09FIXED in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30104/ [01:21:23] Project beta-update-databases-eqiad build #30104: 09FIXED in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30104/ [01:21:26] 10Continuous-Integration-Infrastructure: Depends-On should not work across branches - https://phabricator.wikimedia.org/T210685 (10Legoktm) This isn't due to Depends-On, it's because gate-and-submit-* uses the dependent queue (AFAICT), which means changes are always forced into dependencies like this, just like... [01:21:26] 10Continuous-Integration-Infrastructure: Depends-On should not work across branches - https://phabricator.wikimedia.org/T210685 (10Legoktm) This isn't due to Depends-On, it's because gate-and-submit-* uses the dependent queue (AFAICT), which means changes are always forced into dependencies like this, just like... [03:00:33] 10Continuous-Integration-Config, 10JavaScript, 10MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), 10Patch-For-Review: JS Linting: Drop `dot-notation` keyword over-ride from all repos; no longer needed - https://phabricator.wikimedia.org/T210552 (10Jdforrester-WMF) [03:00:33] 10Continuous-Integration-Config, 10JavaScript, 10MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), 10Patch-For-Review: JS Linting: Drop `dot-notation` keyword over-ride from all repos; no longer needed - https://phabricator.wikimedia.org/T210552 (10Jdforrester-WMF) [03:25:20] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hoo) [03:25:21] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hoo) [04:41:37] 10Continuous-Integration-Infrastructure (Slipway), 10Wikimedia-Fundraising-CiviCRM: Migrate wikimedia-fundraising-civicrm to a Docker container - https://phabricator.wikimedia.org/T210287 (10greg) p:05Triage>03Normal [04:41:37] 10Continuous-Integration-Infrastructure (Slipway), 10Wikimedia-Fundraising-CiviCRM: Migrate wikimedia-fundraising-civicrm to a Docker container - https://phabricator.wikimedia.org/T210287 (10greg) p:05Triage>03Normal [04:42:07] 10Release-Engineering-Team, 10Patch-For-Review: Database busted for CiviCRM tests? - https://phabricator.wikimedia.org/T205950 (10greg) >>! In T205950#4714173, @greg wrote: > (I added this to the 'slipway' CI milestone to indicate that migrating this job to the docker-based system would alleviate this issue (s... [04:42:07] 10Release-Engineering-Team, 10Patch-For-Review: Database busted for CiviCRM tests? - https://phabricator.wikimedia.org/T205950 (10greg) >>! In T205950#4714173, @greg wrote: > (I added this to the 'slipway' CI milestone to indicate that migrating this job to the docker-based system would alleviate this issue (s... [04:42:55] 10Continuous-Integration-Infrastructure (Slipway), 10Wikidata, 10Wikidata Query UI, 10Wikimedia-Portals: Migrate wikidata-query-gui-build and wikimedia-portals-build to Docker containers - https://phabricator.wikimedia.org/T210286 (10greg) p:05Triage>03Normal [04:42:55] 10Continuous-Integration-Infrastructure (Slipway), 10Wikidata, 10Wikidata Query UI, 10Wikimedia-Portals: Migrate wikidata-query-gui-build and wikimedia-portals-build to Docker containers - https://phabricator.wikimedia.org/T210286 (10greg) p:05Triage>03Normal [04:42:57] 10Continuous-Integration-Infrastructure (Slipway), 10Browser-Tests, 10User-zeljkofilipin: Migrate selenium-Wikibase-chrome selenium-WikibaseLexeme-chrome to Docker containers - https://phabricator.wikimedia.org/T210285 (10greg) p:05Triage>03Normal [04:42:57] 10Continuous-Integration-Infrastructure (Slipway), 10Browser-Tests, 10User-zeljkofilipin: Migrate selenium-Wikibase-chrome selenium-WikibaseLexeme-chrome to Docker containers - https://phabricator.wikimedia.org/T210285 (10greg) p:05Triage>03Normal [04:43:03] 10Continuous-Integration-Infrastructure (Slipway): Migrate operations-dns-tabs to a Docker container - https://phabricator.wikimedia.org/T210283 (10greg) p:05Triage>03Normal [04:43:03] 10Continuous-Integration-Infrastructure (Slipway): Migrate operations-dns-tabs to a Docker container - https://phabricator.wikimedia.org/T210283 (10greg) p:05Triage>03Normal [04:43:07] 10Continuous-Integration-Infrastructure (Slipway), 10CirrusSearch, 10Discovery-Search: mwext-CirrusSearch-whitespaces - https://phabricator.wikimedia.org/T210282 (10greg) p:05Triage>03Normal [04:43:07] 10Continuous-Integration-Infrastructure (Slipway), 10CirrusSearch, 10Discovery-Search: mwext-CirrusSearch-whitespaces - https://phabricator.wikimedia.org/T210282 (10greg) p:05Triage>03Normal [04:43:09] 10Continuous-Integration-Infrastructure (Slipway), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: operations-mw-config-typos - https://phabricator.wikimedia.org/T210284 (10greg) p:05Triage>03Normal [04:43:09] 10Continuous-Integration-Infrastructure (Slipway), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: operations-mw-config-typos - https://phabricator.wikimedia.org/T210284 (10greg) p:05Triage>03Normal [04:43:11] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Slipway): Migrate integration-zuul-layoutdiff and integration-zuul-layoutvalidation-gate jobs to Docker containers - https://phabricator.wikimedia.org/T210279 (10greg) p:05Triage>03Normal [04:43:11] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Slipway): Migrate integration-zuul-layoutdiff and integration-zuul-layoutvalidation-gate jobs to Docker containers - https://phabricator.wikimedia.org/T210279 (10greg) p:05Triage>03Normal [04:43:13] 10Continuous-Integration-Infrastructure (Slipway), 10commit-message-validator: Migrate CI job commit-message-validator to a Docker container - https://phabricator.wikimedia.org/T210278 (10greg) p:05Triage>03Normal [04:43:13] 10Continuous-Integration-Infrastructure (Slipway), 10commit-message-validator: Migrate CI job commit-message-validator to a Docker container - https://phabricator.wikimedia.org/T210278 (10greg) p:05Triage>03Normal [04:43:15] 10Continuous-Integration-Infrastructure (Slipway), 10Wikipedia-Android-App-Backlog: Migrate apps/android/wikipedia CI jobs to Docker containers - https://phabricator.wikimedia.org/T210277 (10greg) p:05Triage>03Normal [04:43:15] 10Continuous-Integration-Infrastructure (Slipway), 10Wikipedia-Android-App-Backlog: Migrate apps/android/wikipedia CI jobs to Docker containers - https://phabricator.wikimedia.org/T210277 (10greg) p:05Triage>03Normal [04:43:18] 10Continuous-Integration-Infrastructure (Slipway), 10Wikibugs, 10Patch-For-Review: Migrate CI job labs-tools-wikibugs2-autopull to use a Docker container or phase it out - https://phabricator.wikimedia.org/T210280 (10greg) p:05Triage>03Normal [04:43:18] 10Continuous-Integration-Infrastructure (Slipway), 10Wikibugs, 10Patch-For-Review: Migrate CI job labs-tools-wikibugs2-autopull to use a Docker container or phase it out - https://phabricator.wikimedia.org/T210280 (10greg) p:05Triage>03Normal [04:43:22] 10Continuous-Integration-Infrastructure (Slipway), 10Analytics, 10Analytics-Cluster: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10greg) p:05Triage>03Normal [04:43:22] 10Continuous-Integration-Infrastructure (Slipway), 10Analytics, 10Analytics-Cluster: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10greg) p:05Triage>03Normal [04:43:25] 10Continuous-Integration-Infrastructure (Slipway): Migrate analytics/wikistats CI jobs to Docker - https://phabricator.wikimedia.org/T210272 (10greg) p:05Triage>03Normal [04:43:25] 10Continuous-Integration-Infrastructure (Slipway): Migrate analytics/wikistats CI jobs to Docker - https://phabricator.wikimedia.org/T210272 (10greg) p:05Triage>03Normal [08:34:50] 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 (10hashar) Last rsync entry for `castor02`: Nov 28 08:38:17 castor02 rsyncd[19693]: rsync on caches/pywikibot-core/master/p... [08:34:50] 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 (10hashar) Last rsync entry for `castor02`: Nov 28 08:38:17 castor02 rsyncd[19693]: rsync on caches/pywikibot-core/master/p... [08:35:37] !log Deleted instances castor02 and integration-publishing , replaced by new instances in the new WMCS region | T208803 [08:35:37] !log Deleted instances castor02 and integration-publishing , replaced by new instances in the new WMCS region | T208803 [08:35:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:35:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:35:44] T208803: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 [08:35:45] T208803: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 [08:39:54] PROBLEM - Host castor02 is DOWN: CRITICAL - Host Unreachable (10.68.20.186) [08:39:54] PROBLEM - Host castor02 is DOWN: CRITICAL - Host Unreachable (10.68.20.186) [09:23:59] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10hashar) The last know build was on [[ https://gerrit.wikimedia.org/r/#/c/labs/toollabs/+/474873/ | Gerrit #474873 ]] which triggered both tox... [09:23:59] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10hashar) The last know build was on [[ https://gerrit.wikimedia.org/r/#/c/labs/toollabs/+/474873/ | Gerrit #474873 ]] which triggered both tox... [09:39:03] 10Release-Engineering-Team (Backlog), 10ORES, 10Operations, 10Scoring-platform-team, 10Release Pipeline (Blubber): Blubber should be able to make multi docker files per repo - https://phabricator.wikimedia.org/T210267 (10zeljkofilipin) [09:39:03] 10Release-Engineering-Team (Backlog), 10ORES, 10Operations, 10Scoring-platform-team, 10Release Pipeline (Blubber): Blubber should be able to make multi docker files per repo - https://phabricator.wikimedia.org/T210267 (10zeljkofilipin) [09:39:18] 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10Wikibase-Containers, 10Wikidata, and 2 others: Stretch in docker registry forces ascii encoding - https://phabricator.wikimedia.org/T210260 (10zeljkofilipin) [09:39:18] 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10Wikibase-Containers, 10Wikidata, and 2 others: Stretch in docker registry forces ascii encoding - https://phabricator.wikimedia.org/T210260 (10zeljkofilipin) [09:40:58] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Learn Docker basics - https://phabricator.wikimedia.org/T210412 (10zeljkofilipin) [09:40:58] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Learn Docker basics - https://phabricator.wikimedia.org/T210412 (10zeljkofilipin) [09:46:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10hashar) a:03hashar Fun, if I revert both patches (via [[ https://gerrit.wikimed... [09:46:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10hashar) a:03hashar Fun, if I revert both patches (via [[ https://gerrit.wikimed... [10:02:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10hashar) Comparing the build log. It seems unstable uses python 3.7 now which migh... [10:02:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10hashar) Comparing the build log. It seems unstable uses python 3.7 now which migh... [10:17:16] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [10:17:16] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [10:18:53] (03CR) 10Hashar: [C: 031] "SUCCESS https://integration.wikimedia.org/ci/job/selenium-daily-beta-Newsletter/" [integration/config] - 10https://gerrit.wikimedia.org/r/460555 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [10:18:54] (03CR) 10Hashar: [C: 031] "SUCCESS https://integration.wikimedia.org/ci/job/selenium-daily-beta-Newsletter/" [integration/config] - 10https://gerrit.wikimedia.org/r/460555 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [10:19:35] (03CR) 10Hashar: [C: 031] "SUCCESS https://integration.wikimedia.org/ci/job/selenium-daily-beta-ElectronPdfService/" [integration/config] - 10https://gerrit.wikimedia.org/r/460552 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [10:19:36] (03CR) 10Hashar: [C: 031] "SUCCESS https://integration.wikimedia.org/ci/job/selenium-daily-beta-ElectronPdfService/" [integration/config] - 10https://gerrit.wikimedia.org/r/460552 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [10:20:50] (03CR) 10Hashar: "FAILURE https://integration.wikimedia.org/ci/job/selenium-daily-beta-ORES/" [integration/config] - 10https://gerrit.wikimedia.org/r/460517 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [10:20:50] (03CR) 10Hashar: "FAILURE https://integration.wikimedia.org/ci/job/selenium-daily-beta-ORES/" [integration/config] - 10https://gerrit.wikimedia.org/r/460517 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [10:21:16] (03CR) 10Hashar: "FAILURE https://integration.wikimedia.org/ci/job/selenium-daily-beta-TwoColConflict/" [integration/config] - 10https://gerrit.wikimedia.org/r/460560 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [10:21:16] (03CR) 10Hashar: "FAILURE https://integration.wikimedia.org/ci/job/selenium-daily-beta-TwoColConflict/" [integration/config] - 10https://gerrit.wikimedia.org/r/460560 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [10:21:55] (03CR) 10Hashar: "Missing selenium-daily entry point which is done by https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/AdvancedSearch/+/460395/" [integration/config] - 10https://gerrit.wikimedia.org/r/460516 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [10:21:55] (03CR) 10Hashar: "Missing selenium-daily entry point which is done by https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/AdvancedSearch/+/460395/" [integration/config] - 10https://gerrit.wikimedia.org/r/460516 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [10:27:14] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [10:27:14] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [10:29:43] 10Continuous-Integration-Infrastructure: mwgate-npm-node-6-docker failing with missing ajv/lib/compile/equal - https://phabricator.wikimedia.org/T210712 (10WMDE-Fisch) [10:29:43] 10Continuous-Integration-Infrastructure: mwgate-npm-node-6-docker failing with missing ajv/lib/compile/equal - https://phabricator.wikimedia.org/T210712 (10WMDE-Fisch) [10:31:19] 10Continuous-Integration-Infrastructure: mwgate-npm-node-6-docker failing with missing ajv/lib/compile/equal - https://phabricator.wikimedia.org/T210712 (10Reedy) p:05Triage>03High [10:31:19] 10Continuous-Integration-Infrastructure: mwgate-npm-node-6-docker failing with missing ajv/lib/compile/equal - https://phabricator.wikimedia.org/T210712 (10Reedy) p:05Triage>03High [10:35:17] hashar: ^ it broken :( [10:35:17] hashar: ^ it broken :( [10:36:08] Reedy: who know which dependency started failling ;D [10:36:08] Reedy: who know which dependency started failling ;D [10:36:19] computers suck [10:36:19] computers suck [10:45:35] 10Continuous-Integration-Infrastructure: mwgate-npm-node-6-docker failing with missing ajv/lib/compile/equal - https://phabricator.wikimedia.org/T210712 (10Ladsgroup) I can't merge anything on any extension at the moment, I suggest raising the priority to UBN [10:45:35] 10Continuous-Integration-Infrastructure: mwgate-npm-node-6-docker failing with missing ajv/lib/compile/equal - https://phabricator.wikimedia.org/T210712 (10Ladsgroup) I can't merge anything on any extension at the moment, I suggest raising the priority to UBN [10:45:53] npm suck [10:45:53] npm suck [10:46:27] https://twitter.com/garrows/status/1065217184643768320 [10:46:27] https://twitter.com/garrows/status/1065217184643768320 [10:46:32] Reedy: Amir1 try to find the root cause? It is probably a dependency that changed somehow [10:46:32] Reedy: Amir1 try to find the root cause? It is probably a dependency that changed somehow [10:46:49] hashar: I've been awake about... 18 hours [10:46:49] hashar: I've been awake about... 18 hours [10:46:59] Currently on a flight from Auckland to Doha. I'm going to sleep very soon :P [10:46:59] Currently on a flight from Auckland to Doha. I'm going to sleep very soon :P [10:47:59] hashar: sure. I give it a try (waiting coffee to kicks in) [10:47:59] hashar: sure. I give it a try (waiting coffee to kicks in) [10:54:48] hashar: safe travels [10:54:49] hashar: safe travels [10:55:27] 10Continuous-Integration-Infrastructure: mwgate-npm-node-6-docker failing with missing ajv/lib/compile/equal - https://phabricator.wikimedia.org/T210712 (10hashar) ` 00:00:58.982 ├─┬ eslint-config-wikimedia@0.9.0 00:00:58.983 │ ├─┬ eslint@5.9.0 00:00:58.983 │ │ ├─┬ @babel/code-frame@7.0.0 00:00:58.983 │ │ │ └... [10:55:27] 10Continuous-Integration-Infrastructure: mwgate-npm-node-6-docker failing with missing ajv/lib/compile/equal - https://phabricator.wikimedia.org/T210712 (10hashar) ` 00:00:58.982 ├─┬ eslint-config-wikimedia@0.9.0 00:00:58.983 │ ├─┬ eslint@5.9.0 00:00:58.983 │ │ ├─┬ @babel/code-frame@7.0.0 00:00:58.983 │ │ │ └... [10:55:34] Amir1: https://phabricator.wikimedia.org/T210712#4784637 [10:55:34] Amir1: https://phabricator.wikimedia.org/T210712#4784637 [10:55:47] Amir1: pretty sure that is due to ajv dependency (a json validator) which got a new release like 4 hours ago [10:55:47] Amir1: pretty sure that is due to ajv dependency (a json validator) which got a new release like 4 hours ago [10:56:00] eslint depends on ajv but does not bind to a specific version [10:56:00] eslint depends on ajv but does not bind to a specific version [10:56:44] and eslint is not ours :/ we can pin to older version of eslint (maybe?) [10:56:44] and eslint is not ours :/ we can pin to older version of eslint (maybe?) [11:08:19] 10Phabricator, 10Puppet: Local config file contains escape characters - https://phabricator.wikimedia.org/T103924 (10Aklapper) 05Open>03declined No reply. :( Please reopen this task when clarifying what is the actual problem here and where. (I also assume this is an upstream issue?) Thanks a lot! [11:08:19] 10Phabricator, 10Puppet: Local config file contains escape characters - https://phabricator.wikimedia.org/T103924 (10Aklapper) 05Open>03declined No reply. :( Please reopen this task when clarifying what is the actual problem here and where. (I also assume this is an upstream issue?) Thanks a lot! [11:10:54] 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10Wikibase-Containers, 10Wikidata, and 2 others: Stretch in docker registry forces ascii encoding - https://phabricator.wikimedia.org/T210260 (10LarsWirzenius) Hm, this is strange now. This works on my host: env LC_ALL=C.UTF8 python3 -c "print... [11:10:54] 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10Wikibase-Containers, 10Wikidata, and 2 others: Stretch in docker registry forces ascii encoding - https://phabricator.wikimedia.org/T210260 (10LarsWirzenius) Hm, this is strange now. This works on my host: env LC_ALL=C.UTF8 python3 -c "print... [11:11:38] 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10Wikibase-Containers, 10Wikidata, and 2 others: Stretch in docker registry forces ascii encoding - https://phabricator.wikimedia.org/T210260 (10LarsWirzenius) This does report the C.UTF-8 locale as being available: docker run unitest locale -a [11:11:38] 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10Wikibase-Containers, 10Wikidata, and 2 others: Stretch in docker registry forces ascii encoding - https://phabricator.wikimedia.org/T210260 (10LarsWirzenius) This does report the C.UTF-8 locale as being available: docker run unitest locale -a [11:44:04] hashar: it seems it got automagically fixed [11:44:04] hashar: it seems it got automagically fixed [11:44:21] 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10Wikibase-Containers, 10Wikidata, and 2 others: Stretch in docker registry forces ascii encoding - https://phabricator.wikimedia.org/T210260 (10LarsWirzenius) C.UTF8 does not exist. In every other locale I try, a UTF8 suffix is an alias to the... [11:44:22] 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10Wikibase-Containers, 10Wikidata, and 2 others: Stretch in docker registry forces ascii encoding - https://phabricator.wikimedia.org/T210260 (10LarsWirzenius) C.UTF8 does not exist. In every other locale I try, a UTF8 suffix is an alias to the... [11:54:28] (03CR) 10Zfilipin: [C: 032] Create selenium-daily-beta-Newsletter Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460555 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [11:54:29] (03CR) 10Zfilipin: [C: 032] Create selenium-daily-beta-Newsletter Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460555 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [11:58:03] (03CR) 10Zfilipin: "> Patch Set 3:" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/460517 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [11:58:03] (03CR) 10Zfilipin: "> Patch Set 3:" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/460517 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [12:02:31] 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10Wikibase-Containers, 10Wikidata, and 2 others: Stretch in docker registry forces ascii encoding - https://phabricator.wikimedia.org/T210260 (10LarsWirzenius) If we want to set a default locale for images built by Blubber, we can set LC_ALL or... [12:02:31] 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10Wikibase-Containers, 10Wikidata, and 2 others: Stretch in docker registry forces ascii encoding - https://phabricator.wikimedia.org/T210260 (10LarsWirzenius) If we want to set a default locale for images built by Blubber, we can set LC_ALL or... [12:24:20] Project beta-update-databases-eqiad build #30115: 04FAILURE in 38 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30115/ [12:24:20] Project beta-update-databases-eqiad build #30115: 04FAILURE in 38 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30115/ [12:24:51] 10Beta-Cluster-Infrastructure, 10Puppet: [Cloud VPS alert] Puppet failure on deployment-logstash2.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T210718 (10MarcoAurelio) [12:24:51] 10Beta-Cluster-Infrastructure, 10Puppet: [Cloud VPS alert] Puppet failure on deployment-logstash2.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T210718 (10MarcoAurelio) [12:37:11] Amir1: there is no magic :) [12:37:11] Amir1: there is no magic :) [12:38:45] 10Continuous-Integration-Infrastructure, 10Upstream: mwgate-npm-node-6-docker failing with missing ajv/lib/compile/equal - https://phabricator.wikimedia.org/T210712 (10hashar) 05Open>03Resolved a:03hashar ajv 6.6.1 made its way to npmjs and revert some patch https://github.com/epoberezkin/ajv/commit/bf0f... [12:38:45] 10Continuous-Integration-Infrastructure, 10Upstream: mwgate-npm-node-6-docker failing with missing ajv/lib/compile/equal - https://phabricator.wikimedia.org/T210712 (10hashar) 05Open>03Resolved a:03hashar ajv 6.6.1 made its way to npmjs and revert some patch https://github.com/epoberezkin/ajv/commit/bf0f... [12:38:59] Amir1: https://github.com/epoberezkin/ajv/commit/bf0fc0301e2a61cea17fe1f7b57c100594b846fb :) ajv 6.6.1 has been released in npmjs.org and tha fixed it [12:38:59] Amir1: https://github.com/epoberezkin/ajv/commit/bf0fc0301e2a61cea17fe1f7b57c100594b846fb :) ajv 6.6.1 has been released in npmjs.org and tha fixed it [12:39:12] 10Release-Engineering-Team, 10Operations, 10Scoring-platform-team: Contact number of some WMDE staff should be avalible to SRE/RelEng - https://phabricator.wikimedia.org/T210721 (10Ladsgroup) [12:39:12] 10Release-Engineering-Team, 10Operations, 10Scoring-platform-team: Contact number of some WMDE staff should be avalible to SRE/RelEng - https://phabricator.wikimedia.org/T210721 (10Ladsgroup) [12:42:49] \o/ [12:42:49] \o/ [12:49:46] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10cloud-services-team, 10Puppet: [Cloud VPS alert] Puppet failure on deployment-logstash2.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T210718 (10MarcoAurelio) [12:49:47] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10cloud-services-team, 10Puppet: [Cloud VPS alert] Puppet failure on deployment-logstash2.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T210718 (10MarcoAurelio) [12:55:45] 10Release-Engineering-Team, 10Operations, 10Scoring-platform-team: Contact number of some WMDE staff should be avalible to SRE/RelEng - https://phabricator.wikimedia.org/T210721 (10WMDE-leszek) a:03WMDE-leszek I take it on me. I've briefly talked about this topic with @greg during Technical Conference. We'... [12:55:45] 10Release-Engineering-Team, 10Operations, 10Scoring-platform-team: Contact number of some WMDE staff should be avalible to SRE/RelEng - https://phabricator.wikimedia.org/T210721 (10WMDE-leszek) a:03WMDE-leszek I take it on me. I've briefly talked about this topic with @greg during Technical Conference. We'... [12:55:52] 10Release-Engineering-Team, 10Operations, 10Scoring-platform-team: Contact number of some WMDE staff should be avalible to SRE/RelEng - https://phabricator.wikimedia.org/T210721 (10WMDE-leszek) p:05Triage>03High [12:55:52] 10Release-Engineering-Team, 10Operations, 10Scoring-platform-team: Contact number of some WMDE staff should be avalible to SRE/RelEng - https://phabricator.wikimedia.org/T210721 (10WMDE-leszek) p:05Triage>03High [12:56:34] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10cloud-services-team, 10Puppet: [Cloud VPS alert] Puppet failure on deployment-logstash2.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T210718 (10fgiunchedi) 05Open>03Resolved a:03fgiunchedi Yes this has been fixed by me a few hours ago... [12:56:34] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10cloud-services-team, 10Puppet: [Cloud VPS alert] Puppet failure on deployment-logstash2.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T210718 (10fgiunchedi) 05Open>03Resolved a:03fgiunchedi Yes this has been fixed by me a few hours ago... [12:58:02] I'd like to deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/476228 ok to do so? hashar ? [12:58:02] I'd like to deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/476228 ok to do so? hashar ? [13:09:03] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Learn Docker basics - https://phabricator.wikimedia.org/T210412 (10zeljkofilipin) [13:09:03] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Learn Docker basics - https://phabricator.wikimedia.org/T210412 (10zeljkofilipin) [13:10:16] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Learn Docker basics - https://phabricator.wikimedia.org/T210412 (10zeljkofilipin) [13:10:16] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Learn Docker basics - https://phabricator.wikimedia.org/T210412 (10zeljkofilipin) [13:12:12] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10Patch-For-Review, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10zeljkofilipin) ` gerrit review 12345,45 -m "some message" ` https://gerrit.wikimedia.org/r/Documentation/rest-api-changes.html#set-... [13:12:12] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10Patch-For-Review, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10zeljkofilipin) ` gerrit review 12345,45 -m "some message" ` https://gerrit.wikimedia.org/r/Documentation/rest-api-changes.html#set-... [13:14:13] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10Patch-For-Review, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10zeljkofilipin) [13:14:13] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10Patch-For-Review, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10zeljkofilipin) [13:16:44] hashar: would you have a moment to look at https://gerrit.wikimedia.org/r/c/integration/config/+/475496 ? [13:16:45] hashar: would you have a moment to look at https://gerrit.wikimedia.org/r/c/integration/config/+/475496 ? [13:17:57] seems to be working locally [13:17:57] seems to be working locally [13:17:59] godog: no idea what this patch is doing, that seems to be solely for beta so we can just +2 it at anytime. Usually I announce it via #wikimedia-operations then just rebase on deployment.eqiad.wmnet [13:18:00] godog: no idea what this patch is doing, that seems to be solely for beta so we can just +2 it at anytime. Usually I announce it via #wikimedia-operations then just rebase on deployment.eqiad.wmnet [13:18:22] gehel: too busy for CI patch for now (I am running the train and hotfix). Maybe in an hour or so :/ [13:18:22] gehel: too busy for CI patch for now (I am running the train and hotfix). Maybe in an hour or so :/ [13:18:49] hashar: ack, ping me or kostajh if needed! & thanks! [13:18:49] hashar: ack, ping me or kostajh if needed! & thanks! [13:19:26] hashar: yeah only for beta indeed, the patch is to test sending mw logs through rsyslog -> kafka -> logstash. thanks though I'll just merge it [13:19:26] hashar: yeah only for beta indeed, the patch is to test sending mw logs through rsyslog -> kafka -> logstash. thanks though I'll just merge it [13:21:27] Yippee, build fixed! [13:21:27] Yippee, build fixed! [13:21:27] Project beta-update-databases-eqiad build #30116: 09FIXED in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30116/ [13:21:27] Project beta-update-databases-eqiad build #30116: 09FIXED in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30116/ [13:25:13] 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10Wikibase-Containers, 10Wikidata, and 2 others: Stretch in docker registry forces ascii encoding - https://phabricator.wikimedia.org/T210260 (10akosiaris) >>! In T210260#4784708, @LarsWirzenius wrote: > C.UTF8 does not exist. In every other lo... [13:25:14] 10Release-Engineering-Team (Backlog), 10Scoring-platform-team, 10Wikibase-Containers, 10Wikidata, and 2 others: Stretch in docker registry forces ascii encoding - https://phabricator.wikimedia.org/T210260 (10akosiaris) >>! In T210260#4784708, @LarsWirzenius wrote: > C.UTF8 does not exist. In every other lo... [13:27:47] gehel: ah yeah that patch hmm [13:27:47] gehel: ah yeah that patch hmm [13:30:12] * gehel isn't sure that "hmm" is a good sign :) [13:30:12] * gehel isn't sure that "hmm" is a good sign :) [13:32:23] (03CR) 10Hashar: "I don't think /bin/sh is needed in the ENTRYPOINT (gotta verify)." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/475496 (https://phabricator.wikimedia.org/T209849) (owner: 10Kosta Harlan) [13:32:23] (03CR) 10Hashar: "I don't think /bin/sh is needed in the ENTRYPOINT (gotta verify)." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/475496 (https://phabricator.wikimedia.org/T209849) (owner: 10Kosta Harlan) [13:32:33] gehel: left a couple note, I am building it locally [13:32:33] gehel: left a couple note, I am building it locally [13:32:43] ENTRYPOINT ["/bin/sh", "/opt/sonar-scanner/bin/sonar-scanner"] [13:32:43] ENTRYPOINT ["/bin/sh", "/opt/sonar-scanner/bin/sonar-scanner"] [13:32:47] not sure /bin/sh is needed there [13:32:47] not sure /bin/sh is needed there [13:33:04] kostajh: ^^ :) [13:33:04] kostajh: ^^ :) [13:33:12] sorry that takes a while, got too many other things to do [13:33:12] sorry that takes a while, got too many other things to do [13:33:15] hashar: thanks, responding now [13:33:15] hashar: thanks, responding now [13:33:57] and gotta create an entry in the changelog file [13:33:57] and gotta create an entry in the changelog file [13:34:11] (03CR) 10Kosta Harlan: Fix Dockerfile.template for sonar-scanner (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/475496 (https://phabricator.wikimedia.org/T209849) (owner: 10Kosta Harlan) [13:34:11] (03CR) 10Kosta Harlan: Fix Dockerfile.template for sonar-scanner (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/475496 (https://phabricator.wikimedia.org/T209849) (owner: 10Kosta Harlan) [13:36:08] (03CR) 10Hashar: Fix Dockerfile.template for sonar-scanner (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/475496 (https://phabricator.wikimedia.org/T209849) (owner: 10Kosta Harlan) [13:36:08] (03CR) 10Hashar: Fix Dockerfile.template for sonar-scanner (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/475496 (https://phabricator.wikimedia.org/T209849) (owner: 10Kosta Harlan) [13:36:28] kostajh: looks like we can drop the /bin/sh entirely since the ENTRYPOINT no more rely on a variable [13:36:28] kostajh: looks like we can drop the /bin/sh entirely since the ENTRYPOINT no more rely on a variable [13:36:38] then add an entry in changelog and I will +2 it :) [13:36:38] then add an entry in changelog and I will +2 it :) [13:36:40] (03PS6) 10Kosta Harlan: Fix Dockerfile.template for sonar-scanner [integration/config] - 10https://gerrit.wikimedia.org/r/475496 (https://phabricator.wikimedia.org/T209849) [13:36:40] (03PS6) 10Kosta Harlan: Fix Dockerfile.template for sonar-scanner [integration/config] - 10https://gerrit.wikimedia.org/r/475496 (https://phabricator.wikimedia.org/T209849) [13:37:34] hashar: do we need a bump of the version? 0.1.0 was never built [13:37:34] hashar: do we need a bump of the version? 0.1.0 was never built [13:38:01] oh [13:38:01] oh [13:38:05] yeah because it was broken wasn't it ? [13:38:05] yeah because it was broken wasn't it ? [13:38:07] I can do it of course [13:38:07] I can do it of course [13:38:08] yeah [13:38:08] yeah [13:38:50] (03CR) 10Hashar: [C: 032] "From Kosta: the container could not be build, so there is no need to bump the changelog." [integration/config] - 10https://gerrit.wikimedia.org/r/475496 (https://phabricator.wikimedia.org/T209849) (owner: 10Kosta Harlan) [13:38:50] (03CR) 10Hashar: [C: 032] "From Kosta: the container could not be build, so there is no need to bump the changelog." [integration/config] - 10https://gerrit.wikimedia.org/r/475496 (https://phabricator.wikimedia.org/T209849) (owner: 10Kosta Harlan) [13:38:54] easier ;) [13:38:54] easier ;) [13:39:05] hashar++ [13:39:05] hashar++ [13:39:09] thank you [13:39:09] thank you [13:40:31] (03Merged) 10jenkins-bot: Fix Dockerfile.template for sonar-scanner [integration/config] - 10https://gerrit.wikimedia.org/r/475496 (https://phabricator.wikimedia.org/T209849) (owner: 10Kosta Harlan) [13:40:31] (03Merged) 10jenkins-bot: Fix Dockerfile.template for sonar-scanner [integration/config] - 10https://gerrit.wikimedia.org/r/475496 (https://phabricator.wikimedia.org/T209849) (owner: 10Kosta Harlan) [13:53:52] building it now [13:53:52] building it now [13:54:33] !log Building docker-registry.discovery.wmnet/releng/java8-sonar-scanner:0.1.0 for https://gerrit.wikimedia.org/r/475496 | T209849 [13:54:33] !log Building docker-registry.discovery.wmnet/releng/java8-sonar-scanner:0.1.0 for https://gerrit.wikimedia.org/r/475496 | T209849 [13:54:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:54:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:54:36] T209849: Create a docker container with sonar-scanner for CI - https://phabricator.wikimedia.org/T209849 [13:54:36] T209849: Create a docker container with sonar-scanner for CI - https://phabricator.wikimedia.org/T209849 [14:00:19] gehel: kostajh image build at least! [14:00:19] gehel: kostajh image build at least! [14:00:28] hashar: thanks! [14:00:28] hashar: thanks! [14:01:11] hashar: cool! `docker pull docker-registry.wikimedia.org/releng/java8-sonar-scanner:0.1.0` does not yet work, do I need to wait longer? [14:01:11] hashar: cool! `docker pull docker-registry.wikimedia.org/releng/java8-sonar-scanner:0.1.0` does not yet work, do I need to wait longer? [14:01:29] should be instance [14:01:30] should be instance [14:01:32] instant [14:01:32] instant [14:01:41] unless there is some cache layer in between internet and the registry :( [14:01:41] unless there is some cache layer in between internet and the registry :( [14:02:38] yeah must be cache related :( [14:02:38] yeah must be cache related :( [14:06:04] ok [14:06:04] ok [14:09:30] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) wikidatawiki is back to 1.33.0-wmf.6 [14:09:31] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) wikidatawiki is back to 1.33.0-wmf.6 [14:26:24] !log switch mediawiki logging to use localhost syslog -> kafka -> logstash [14:26:24] !log switch mediawiki logging to use localhost syslog -> kafka -> logstash [14:26:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:26:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:30:23] hashar: yay, just pulled it! [14:30:23] hashar: yay, just pulled it! [14:46:21] 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 (10Andrew) Thanks! [14:46:21] 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 (10Andrew) Thanks! [14:46:31] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [14:46:31] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [14:54:08] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [14:54:09] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [15:05:20] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [15:05:21] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [15:13:17] yoohooooo :) [15:13:17] yoohooooo :) [15:13:25] looks like I lost some Jenkins credentials somehow? [15:13:25] looks like I lost some Jenkins credentials somehow? [15:13:31] i used to be able to access https://integration.wikimedia.org/ci/credentials/ [15:13:31] i used to be able to access https://integration.wikimedia.org/ci/credentials/ [15:13:36] now i get Ottomata is missing the Credentials/View permission [15:13:36] now i get Ottomata is missing the Credentials/View permission [15:13:49] and for https://integration.wikimedia.org/ci/configfiles/ [15:13:49] and for https://integration.wikimedia.org/ci/configfiles/ [15:13:52] Ottomata is missing the Job/ExtendedRead permission [15:13:52] Ottomata is missing the Job/ExtendedRead permission [15:15:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10Bstorm) The python code had issues on python 3.7 in general when I messed with it... [15:15:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10Bstorm) The python code had issues on python 3.7 in general when I messed with it... [15:25:57] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [15:25:57] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [15:46:27] ottomata: I don't have them too :) [15:46:27] ottomata: I don't have them too :) [15:47:12] I need to change archiva-ci's password in there [15:47:12] I need to change archiva-ci's password in there [15:48:26] anybody can help? [15:48:26] anybody can help? [15:48:58] how do I add the permission? I have admin in Jenkins [15:48:58] how do I add the permission? I have admin in Jenkins [15:56:20] greg-g: o/ - can I drop the password in your home somewhere if you want [15:56:20] greg-g: o/ - can I drop the password in your home somewhere if you want [15:56:27] *I can [15:56:27] *I can [15:56:50] I'd need to update archiva-ci's pass only [15:56:50] I'd need to update archiva-ci's pass only [16:10:00] elukey: sure? :) [16:10:00] elukey: sure? :) [16:10:17] a deployment server would be ok [16:10:17] a deployment server would be ok [16:32:25] greg-g: sorry for the lag - I just placed the pass in /home/gjg/archiva_pass.txt on deploy1001 [16:32:25] greg-g: sorry for the lag - I just placed the pass in /home/gjg/archiva_pass.txt on deploy1001 [16:35:54] 10Beta-Cluster-Infrastructure, 10Readers-Web-Backlog: [Bug] Beta cluster reports 500 for mobile pages - https://phabricator.wikimedia.org/T210748 (10Niedzielski) [16:35:55] 10Beta-Cluster-Infrastructure, 10Readers-Web-Backlog: [Bug] Beta cluster reports 500 for mobile pages - https://phabricator.wikimedia.org/T210748 (10Niedzielski) [16:40:15] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10cloud-services-team, 10Puppet: [Cloud VPS alert] Puppet failure on deployment-logstash2.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T210718 (10MarcoAurelio) Thanks :) [16:40:15] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10cloud-services-team, 10Puppet: [Cloud VPS alert] Puppet failure on deployment-logstash2.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T210718 (10MarcoAurelio) Thanks :) [16:40:47] elukey: for archiva-deploy-user ? [16:40:47] elukey: for archiva-deploy-user ? [16:41:16] oh, no: archiva-ci/****** (Archiva user for automated releases) [16:41:16] oh, no: archiva-ci/****** (Archiva user for automated releases) [16:43:16] elukey: done, let me know if it works! [16:43:16] elukey: done, let me know if it works! [16:45:45] (03PS1) 10Thcipriani: service-pipline: reformat date tag [integration/config] - 10https://gerrit.wikimedia.org/r/476547 (https://phabricator.wikimedia.org/T209088) [16:45:45] (03PS1) 10Thcipriani: service-pipline: reformat date tag [integration/config] - 10https://gerrit.wikimedia.org/r/476547 (https://phabricator.wikimedia.org/T209088) [16:49:52] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Wikimedia-Logstash, 10Patch-For-Review, 10Technical-Debt: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324 (10herron) [16:49:52] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Wikimedia-Logstash, 10Patch-For-Review, 10Technical-Debt: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324 (10herron) [16:53:20] greg-g: thank youuuu [16:53:20] greg-g: thank youuuu [17:56:45] (03PS1) 10Thcipriani: service-pipeline: trigger the correct jobs [integration/config] - 10https://gerrit.wikimedia.org/r/476580 [17:56:45] (03PS1) 10Thcipriani: service-pipeline: trigger the correct jobs [integration/config] - 10https://gerrit.wikimedia.org/r/476580 [18:00:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10Bstorm) There! I made it work with python 3.7. I didn't want to change the rele... [18:00:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10Bstorm) There! I made it work with python 3.7. I didn't want to change the rele... [18:07:43] (03CR) 10Thcipriani: [C: 032] "deployed and verified" [integration/config] - 10https://gerrit.wikimedia.org/r/476580 (owner: 10Thcipriani) [18:07:43] (03CR) 10Thcipriani: [C: 032] "deployed and verified" [integration/config] - 10https://gerrit.wikimedia.org/r/476580 (owner: 10Thcipriani) [18:10:14] (03Merged) 10jenkins-bot: service-pipeline: trigger the correct jobs [integration/config] - 10https://gerrit.wikimedia.org/r/476580 (owner: 10Thcipriani) [18:10:14] (03Merged) 10jenkins-bot: service-pipeline: trigger the correct jobs [integration/config] - 10https://gerrit.wikimedia.org/r/476580 (owner: 10Thcipriani) [18:13:03] (03PS2) 10Thcipriani: service-pipeline: reformat date tag [integration/config] - 10https://gerrit.wikimedia.org/r/476547 (https://phabricator.wikimedia.org/T209088) [18:13:03] (03PS2) 10Thcipriani: service-pipeline: reformat date tag [integration/config] - 10https://gerrit.wikimedia.org/r/476547 (https://phabricator.wikimedia.org/T209088) [18:21:12] (03CR) 10Lars Wirzenius: [C: 031] "Looks good to me!" [integration/config] - 10https://gerrit.wikimedia.org/r/476547 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [18:21:12] (03CR) 10Lars Wirzenius: [C: 031] "Looks good to me!" [integration/config] - 10https://gerrit.wikimedia.org/r/476547 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [18:29:43] (03PS3) 10Thcipriani: service-pipeline: reformat date tag [integration/config] - 10https://gerrit.wikimedia.org/r/476547 (https://phabricator.wikimedia.org/T209088) [18:29:43] (03PS3) 10Thcipriani: service-pipeline: reformat date tag [integration/config] - 10https://gerrit.wikimedia.org/r/476547 (https://phabricator.wikimedia.org/T209088) [18:29:45] (03PS1) 10Thcipriani: service-pipeline: add commit SHA tag to images [integration/config] - 10https://gerrit.wikimedia.org/r/476593 (https://phabricator.wikimedia.org/T209088) [18:29:45] (03PS1) 10Thcipriani: service-pipeline: add commit SHA tag to images [integration/config] - 10https://gerrit.wikimedia.org/r/476593 (https://phabricator.wikimedia.org/T209088) [18:32:32] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10greg) [18:32:33] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10greg) [18:33:37] (03CR) 10Thcipriani: [C: 032] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/476547 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [18:33:37] (03CR) 10Thcipriani: [C: 032] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/476547 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [18:36:02] (03Merged) 10jenkins-bot: service-pipeline: reformat date tag [integration/config] - 10https://gerrit.wikimedia.org/r/476547 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [18:36:02] (03Merged) 10jenkins-bot: service-pipeline: reformat date tag [integration/config] - 10https://gerrit.wikimedia.org/r/476547 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [18:36:10] (03CR) 10Lars Wirzenius: [C: 031] "The actual change" [integration/config] - 10https://gerrit.wikimedia.org/r/476593 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [18:36:10] (03CR) 10Lars Wirzenius: [C: 031] "The actual change" [integration/config] - 10https://gerrit.wikimedia.org/r/476593 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [18:46:55] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Browser-Tests, and 2 others: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10Jdlrobson) p:05High>03Unbreak! [18:46:55] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Browser-Tests, and 2 others: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10Jdlrobson) p:05High>03Unbreak! [18:49:16] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10g... [18:49:16] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10g... [18:50:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10J... [18:50:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10J... [18:51:24] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10J... [18:51:24] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10J... [18:54:54] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10Anomie) [18:54:54] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10Anomie) [18:56:10] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10J... [18:56:10] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10J... [18:56:23] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10g... [18:56:23] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10g... [18:57:58] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10J... [18:57:58] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10J... [18:58:56] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10g... [18:58:56] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10g... [18:59:36] thanks greg-g this beta cluster problem has been stressing me out all this week. Glad we've finally worked out what's going on <3 [18:59:36] thanks greg-g this beta cluster problem has been stressing me out all this week. Glad we've finally worked out what's going on <3 [19:01:04] sorry :( [19:01:04] sorry :( [19:11:30] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10K... [19:11:31] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10K... [19:17:52] (03PS2) 10Thcipriani: service-pipeline: add commit SHA tag to images [integration/config] - 10https://gerrit.wikimedia.org/r/476593 (https://phabricator.wikimedia.org/T209088) [19:17:52] (03PS2) 10Thcipriani: service-pipeline: add commit SHA tag to images [integration/config] - 10https://gerrit.wikimedia.org/r/476593 (https://phabricator.wikimedia.org/T209088) [19:17:54] (03PS1) 10Thcipriani: service-pipeline: add image tag on git tag push [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) [19:17:54] (03PS1) 10Thcipriani: service-pipeline: add image tag on git tag push [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) [19:20:24] Krenair: that would be good to do, the tone could improve. [19:20:24] Krenair: that would be good to do, the tone could improve. [19:21:31] Project beta-update-databases-eqiad build #30122: 04FAILURE in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30122/ [19:21:31] Project beta-update-databases-eqiad build #30122: 04FAILURE in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30122/ [19:23:02] (03PS2) 10Thcipriani: service-pipeline: add image tag on git tag push [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) [19:23:02] (03PS2) 10Thcipriani: service-pipeline: add image tag on git tag push [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) [19:25:05] (03CR) 10Lars Wirzenius: [C: 031] "Looks OK to me. The commit message could perhaps clarify that the old tag (the production tag) is still there. For dumpkompfen like me." [integration/config] - 10https://gerrit.wikimedia.org/r/476593 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [19:25:05] (03CR) 10Lars Wirzenius: [C: 031] "Looks OK to me. The commit message could perhaps clarify that the old tag (the production tag) is still there. For dumpkompfen like me." [integration/config] - 10https://gerrit.wikimedia.org/r/476593 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [19:27:29] (03CR) 10Lars Wirzenius: [C: 031] "Commit message verb: "will populated"?" [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [19:27:29] (03CR) 10Lars Wirzenius: [C: 031] "Commit message verb: "will populated"?" [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [19:31:35] (03PS3) 10Thcipriani: service-pipeline: add commit SHA tag to images [integration/config] - 10https://gerrit.wikimedia.org/r/476593 (https://phabricator.wikimedia.org/T209088) [19:31:35] (03PS3) 10Thcipriani: service-pipeline: add commit SHA tag to images [integration/config] - 10https://gerrit.wikimedia.org/r/476593 (https://phabricator.wikimedia.org/T209088) [19:31:37] (03PS3) 10Thcipriani: service-pipeline: add image tag on git tag push [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) [19:31:37] (03PS3) 10Thcipriani: service-pipeline: add image tag on git tag push [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) [19:40:04] (03CR) 10Lars Wirzenius: [V: 04-1 C: 031] service-pipeline: add image tag on git tag push [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [19:40:04] (03CR) 10Lars Wirzenius: [V: 04-1 C: 031] service-pipeline: add image tag on git tag push [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [19:41:04] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [19:41:04] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [19:51:48] 10Release-Engineering-Team (Kanban), 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10hashar) 05Open>03Resolved [19:51:48] 10Release-Engineering-Team (Kanban), 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10hashar) 05Open>03Resolved [19:56:58] 10Continuous-Integration-Config: labs/toollabs should always run debian-glue, possibly with multiple distributions - https://phabricator.wikimedia.org/T210780 (10hashar) [19:56:58] 10Continuous-Integration-Config: labs/toollabs should always run debian-glue, possibly with multiple distributions - https://phabricator.wikimedia.org/T210780 (10hashar) [19:57:23] 10Release-Engineering-Team (Kanban), 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10hashar) I have filled T210780 to tweak the CI configuration. [19:57:23] 10Release-Engineering-Team (Kanban), 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10hashar) I have filled T210780 to tweak the CI configuration. [20:01:08] 10Beta-Cluster-Infrastructure, 10Readers-Web-Backlog: [Bug] Beta cluster reports 500 for mobile pages - https://phabricator.wikimedia.org/T210748 (10Jdlrobson) [20:01:09] 10Beta-Cluster-Infrastructure, 10Readers-Web-Backlog: [Bug] Beta cluster reports 500 for mobile pages - https://phabricator.wikimedia.org/T210748 (10Jdlrobson) [20:01:11] 10Beta-Cluster-Infrastructure, 10Operations: "Obama" page on Beta Cluster often responds with 503 - https://phabricator.wikimedia.org/T188913 (10Jdlrobson) [20:01:11] 10Beta-Cluster-Infrastructure, 10Operations: "Obama" page on Beta Cluster often responds with 503 - https://phabricator.wikimedia.org/T188913 (10Jdlrobson) [20:09:58] thcipriani: I could use your logstash blessing :) I am trying to find past occurences of logs for https://phabricator.wikimedia.org/T210739 [20:09:58] thcipriani: I could use your logstash blessing :) I am trying to find past occurences of logs for https://phabricator.wikimedia.org/T210739 [20:10:30] namely, I am looking for entries that have message matching Failed to delete page-move revision [20:10:30] namely, I am looking for entries that have message matching Failed to delete page-move revision [20:10:34] but can't find a way :( [20:10:34] but can't find a way :( [20:10:39] Krenair: looking into the read-only issue. any idea if seconds_behind_master is logged anywhere? [20:10:40] Krenair: looking into the read-only issue. any idea if seconds_behind_master is logged anywhere? [20:10:48] i can't find it in grafana [20:10:48] i can't find it in grafana [20:11:12] marxarelli, not aware of it [20:11:12] marxarelli, not aware of it [20:11:31] hmm [20:11:31] hmm [20:11:51] i know beta's prometheus collector invokes the mysql collector... [20:11:51] i know beta's prometheus collector invokes the mysql collector... [20:11:52] you could quickly set up something that periodically runs sudo -i mysql -e 'show slave status \G' | grep Seconds_Behind_Master [20:11:53] you could quickly set up something that periodically runs sudo -i mysql -e 'show slave status \G' | grep Seconds_Behind_Master [20:12:01] dumps the result to a file [20:12:01] dumps the result to a file [20:12:02] godog: ^ do you know? [20:12:02] godog: ^ do you know? [20:12:06] hashar: this found a few https://logstash.wikimedia.org/goto/2a9dc287296fc68f61a22d19967a06a5 matching against that message for time period of Today [20:12:06] hashar: this found a few https://logstash.wikimedia.org/goto/2a9dc287296fc68f61a22d19967a06a5 matching against that message for time period of Today [20:13:00] hmm [20:13:00] hmm [20:13:23] I have tried in the search box [20:13:23] I have tried in the search box [20:13:30] hashar: 14 instances in the past 6 or 7 hours [20:13:30] hashar: 14 instances in the past 6 or 7 hours [20:13:34] with message:"Failed to delete page-move revision" but that gives nothing [20:13:34] with message:"Failed to delete page-move revision" but that gives nothing [20:14:23] * * * * * date >> /home/krenair/replag.log; sudo -i mysql -e 'show slave status \G' | grep Seconds_Behind_Master >> /home/krenair/replag.log [20:14:24] * * * * * date >> /home/krenair/replag.log; sudo -i mysql -e 'show slave status \G' | grep Seconds_Behind_Master >> /home/krenair/replag.log [20:14:28] feel free to go tail that file marxarelli ^ [20:14:28] feel free to go tail that file marxarelli ^ [20:14:42] :) thanks! [20:14:42] :) thanks! [20:14:43] hashar: yeah, the search box query and the bubbles are different I've noticed. The bubbles seem like they are more likely to show me what I want, usually. [20:14:43] hashar: yeah, the search box query and the bubbles are different I've noticed. The bubbles seem like they are more likely to show me what I want, usually. [20:15:00] and now it is working ... [20:15:00] and now it is working ... [20:16:19] I'm a little bit confused by why AbuseFilter CI is failing for REL branches (and presumably all extensions in Gerrit that use a composer library that is not needed by core/tarball). Afaik this used to work, do we know what broke it? ref. https://phabricator.wikimedia.org/T189560 [20:16:19] I'm a little bit confused by why AbuseFilter CI is failing for REL branches (and presumably all extensions in Gerrit that use a composer library that is not needed by core/tarball). Afaik this used to work, do we know what broke it? ref. https://phabricator.wikimedia.org/T189560 [20:18:12] What I mean is.. can we still restore the way it previously worked as interim solution? [20:18:12] What I mean is.. can we still restore the way it previously worked as interim solution? [20:18:30] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [20:18:30] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [20:18:51] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) T210739 occurred previously, not a blocker anymore. [20:18:51] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) T210739 occurred previously, not a blocker anymore. [20:19:02] thcipriani: I found a wmf.4 message so that is not a new issue. Thank you [20:19:02] thcipriani: I found a wmf.4 message so that is not a new issue. Thank you [20:19:22] hashar: happy to help! [20:19:22] hashar: happy to help! [20:19:42] Krinkle: I havent looked into it nor do i have any solution :-/ [20:19:42] Krinkle: I havent looked into it nor do i have any solution :-/ [20:20:09] but in short, we need to overhaul how we test mediawiki. There are too many use cases [20:20:09] but in short, we need to overhaul how we test mediawiki. There are too many use cases [20:21:20] Yippee, build fixed! [20:21:20] Yippee, build fixed! [20:21:21] Project beta-update-databases-eqiad build #30123: 09FIXED in 1 min 20 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30123/ [20:21:21] Project beta-update-databases-eqiad build #30123: 09FIXED in 1 min 20 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30123/ [20:23:24] * hashar tries logstash search for (message:"GeoCrumbs::getParentRegion" AND type:hhvm) [20:23:24] * hashar tries logstash search for (message:"GeoCrumbs::getParentRegion" AND type:hhvm) [20:24:38] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10d... [20:24:38] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10d... [20:25:55] hashar: Hm.. I'm sure this worked before though. I suspect the splitting/cleaning of vendor from quibble might be causing. E.g. previously it would run composer install/test in the exxtension dir first, at which point it works fine. [20:25:55] hashar: Hm.. I'm sure this worked before though. I suspect the splitting/cleaning of vendor from quibble might be causing. E.g. previously it would run composer install/test in the exxtension dir first, at which point it works fine. [20:26:00] It pulls in the dependencies as needed. [20:26:00] It pulls in the dependencies as needed. [20:31:11] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10d... [20:31:11] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10d... [20:33:25] marxarelli, I think MediaWiki uses Seconds_Behind_Master by default [20:33:25] marxarelli, I think MediaWiki uses Seconds_Behind_Master by default [20:33:42] it can be configured to use some heartbeat thing but I don't think we have that in beta [20:33:42] it can be configured to use some heartbeat thing but I don't think we have that in beta [20:35:16] right on. checking out LoadBalancer.php atm... [20:35:16] right on. checking out LoadBalancer.php atm... [20:35:28] see includes/libs/rdbms/database/DatabaseMysqlBase.php [20:35:28] see includes/libs/rdbms/database/DatabaseMysqlBase.php [20:35:32] getLag function [20:35:33] getLag function [20:36:00] if I'm right it'll be using getLagFromSlaveStatus which literally just does show slave status [20:36:00] if I'm right it'll be using getLagFromSlaveStatus which literally just does show slave status [20:36:06] and then checks Seconds_Behind_Master in the result [20:36:06] and then checks Seconds_Behind_Master in the result [20:36:37] does it check all configured dbs, or just non-masters? [20:36:37] does it check all configured dbs, or just non-masters? [20:36:55] good question not sure [20:36:55] good question not sure [20:37:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10d... [20:37:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode - https://phabricator.wikimedia.org/T210557 (10d... [20:38:26] Database.php's getApproximateLagStatus checks whether it's a replica before checking lag [20:38:26] Database.php's getApproximateLagStatus checks whether it's a replica before checking lag [20:38:34] would be interesting to find out whether the replica status is accurate [20:38:35] would be interesting to find out whether the replica status is accurate [20:38:43] (03PS4) 10Thcipriani: service-pipeline: add image tag on git tag push [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) [20:38:43] (03PS4) 10Thcipriani: service-pipeline: add image tag on git tag push [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) [20:38:45] has wikibugs always truncated in this fashion? [20:38:46] has wikibugs always truncated in this fashion? [20:39:20] I don't recall seeing that before [20:39:20] I don't recall seeing that before [20:43:25] yeah, i wonder what the replica status is based on [20:43:25] yeah, i wonder what the replica status is based on [20:51:01] 10Gerrit: Unable to change full name in Gerrit through contact settings page or command line - https://phabricator.wikimedia.org/T206129 (10Paladox) [20:51:01] 10Gerrit: Unable to change full name in Gerrit through contact settings page or command line - https://phabricator.wikimedia.org/T206129 (10Paladox) [20:51:55] 10Gerrit, 10Upstream: PolyGerrit doesn't support old UI URLs - https://phabricator.wikimedia.org/T196895 (10Paladox) [20:51:55] 10Gerrit, 10Upstream: PolyGerrit doesn't support old UI URLs - https://phabricator.wikimedia.org/T196895 (10Paladox) [20:52:15] 10Gerrit, 10Patch-For-Review: Upgrade to Gerrit 2.16 - https://phabricator.wikimedia.org/T200739 (10Paladox) [20:52:15] 10Gerrit, 10Patch-For-Review: Upgrade to Gerrit 2.16 - https://phabricator.wikimedia.org/T200739 (10Paladox) [20:52:17] 10Gerrit, 10Upstream: PolyGerrit doesn't support old UI URLs - https://phabricator.wikimedia.org/T196895 (10Paladox) [20:52:18] 10Gerrit, 10Upstream: PolyGerrit doesn't support old UI URLs - https://phabricator.wikimedia.org/T196895 (10Paladox) [20:52:19] Krinkle: are you still around? wikimedia-production-error board is awesome, I am wondering if we could get a column for config changes :) [20:52:19] Krinkle: are you still around? wikimedia-production-error board is awesome, I am wondering if we could get a column for config changes :) [20:52:34] What kind of config? [20:52:34] What kind of config? [20:52:41] Krinkle: the context is the deployment of the actor feature which causes log but is not tight to a specific wmf version [20:52:41] Krinkle: the context is the deployment of the actor feature which causes log but is not tight to a specific wmf version [20:53:03] the epic task being https://phabricator.wikimedia.org/T188327 "Deploy refactored actor storage" [20:53:03] the epic task being https://phabricator.wikimedia.org/T188327 "Deploy refactored actor storage" [20:53:05] hashar: columns are weeks [20:53:05] hashar: columns are weeks [20:53:08] this week is wmf.6 [20:53:08] this week is wmf.6 [20:53:41] ahh [20:53:41] ahh [20:53:41] I can add ISO numbers to them if it helps. I started with that, but figured people would prefer branch names. [20:53:41] I can add ISO numbers to them if it helps. I started with that, but figured people would prefer branch names. [20:53:51] na na thanks :) [20:53:51] na na thanks :) [20:54:04] It's not very intuitive for non-MW, I agree :) [20:54:04] It's not very intuitive for non-MW, I agree :) [20:54:06] in my mind it was the mediawiki version and thus tied to mediawiki itself [20:54:06] in my mind it was the mediawiki version and thus tied to mediawiki itself [20:54:18] weeks sounds good :) thanks! [20:54:18] weeks sounds good :) thanks! [20:55:21] Krinkle: and Tyler has shown me the form to report application errors. Well done! [20:55:21] Krinkle: and Tyler has shown me the form to report application errors. Well done! [20:55:36] we shuld somehow add a link to it from the train task template [20:55:36] we shuld somehow add a link to it from the train task template [20:56:09] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [20:56:09] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [20:56:34] 10Gerrit: context links in diff don't use pointer cursor - https://phabricator.wikimedia.org/T210515 (10Paladox) his has been fixed upstream (in the 2.16 release) and also the colour has been updated. [20:56:34] 10Gerrit: context links in diff don't use pointer cursor - https://phabricator.wikimedia.org/T210515 (10Paladox) his has been fixed upstream (in the 2.16 release) and also the colour has been updated. [20:56:48] 10Gerrit, 10Patch-For-Review: Upgrade to Gerrit 2.16 - https://phabricator.wikimedia.org/T200739 (10Paladox) [20:56:48] 10Gerrit, 10Patch-For-Review: Upgrade to Gerrit 2.16 - https://phabricator.wikimedia.org/T200739 (10Paladox) [20:56:53] 10Gerrit: context links in diff don't use pointer cursor - https://phabricator.wikimedia.org/T210515 (10Paladox) [20:56:53] 10Gerrit: context links in diff don't use pointer cursor - https://phabricator.wikimedia.org/T210515 (10Paladox) [20:57:13] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) T210621 is no more a blocker, it is related to the Actor deployment and a config flag that got changed. [20:57:14] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) T210621 is no more a blocker, it is related to the Actor deployment and a config flag that got changed. [20:59:33] 10Gerrit: Gerrit's "Show Diffs" button does not always expand all diffs for each changed file - https://phabricator.wikimedia.org/T199012 (10Paladox) i *think* this may be fixed in 2.16. [20:59:33] 10Gerrit: Gerrit's "Show Diffs" button does not always expand all diffs for each changed file - https://phabricator.wikimedia.org/T199012 (10Paladox) i *think* this may be fixed in 2.16. [21:00:05] hashar: I went back in time and recommended that to mmodel, who has done and deployed that :) [21:00:05] hashar: I went back in time and recommended that to mmodel, who has done and deployed that :) [21:00:40] Its the little link on "Use this form to create one." about 2/3 down the Blockers Task details generated text. [21:00:40] Its the little link on "Use this form to create one." about 2/3 down the Blockers Task details generated text. [21:04:10] Krinkle: OHH [21:04:28] Krinkle: I should have read the actual task. Very handy thanks [21:04:35] Krinkle: OHH [21:04:35] Krinkle: I should have read the actual task. Very handy thanks [21:18:47] hashar: yw - btw, at TechConf chatting with Adam, I noticed the create forms are also on the workboard dropdown menus [21:18:47] hashar: yw - btw, at TechConf chatting with Adam, I noticed the create forms are also on the workboard dropdown menus [21:18:48] https://i.imgur.com/BN4wD1Z.png [21:18:48] https://i.imgur.com/BN4wD1Z.png [21:19:00] Which is another way to create a task with the form, -and- in the right column [21:19:00] Which is another way to create a task with the form, -and- in the right column [21:20:19] Krinkle: yeah that is what tyler has shown to me. Very handy [21:20:19] Krinkle: yeah that is what tyler has shown to me. Very handy [21:20:28] ah cool! [21:20:28] ah cool! [21:20:50] and kudos on the magic logstash that filter out old logs [21:20:50] and kudos on the magic logstash that filter out old logs [21:21:18] I am sure we could use some kind of machine learning to process them more or less automatically [21:21:18] I am sure we could use some kind of machine learning to process them more or less automatically [21:32:42] (03CR) 10Jeena Huneidi: [C: 031] "Thanks for the walk through!" [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [21:32:43] (03CR) 10Jeena Huneidi: [C: 031] "Thanks for the walk through!" [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [21:51:42] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [21:51:42] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [21:52:25] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) GeoCrumbs fatal was a one off spike and an issue in the code that predate 1.33.0-wmf.6 T210734 is no more a blocker. [21:52:25] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) GeoCrumbs fatal was a one off spike and an issue in the code that predate 1.33.0-wmf.6 T210734 is no more a blocker. [22:20:34] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) 05Open>03Resolved Should be stable now. All blockers got quickly fixed, some other turned out to be pre existing or one off is... [22:20:34] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) 05Open>03Resolved Should be stable now. All blockers got quickly fixed, some other turned out to be pre existing or one off is... [22:44:39] (03CR) 10Hashar: "I am not sure ZUUL_COMMIT is suitable for publishing. If one CR+2 a change that is not directly on tip of the branch, Zuul crafts a merge " [integration/config] - 10https://gerrit.wikimedia.org/r/476593 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [22:44:39] (03CR) 10Hashar: "I am not sure ZUUL_COMMIT is suitable for publishing. If one CR+2 a change that is not directly on tip of the branch, Zuul crafts a merge " [integration/config] - 10https://gerrit.wikimedia.org/r/476593 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [22:46:15] (03CR) 10Hashar: "Also, if we rebuild for a given commit, the resulting image will be different since the build is not reproducible :)" [integration/config] - 10https://gerrit.wikimedia.org/r/476593 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [22:46:15] (03CR) 10Hashar: "Also, if we rebuild for a given commit, the resulting image will be different since the build is not reproducible :)" [integration/config] - 10https://gerrit.wikimedia.org/r/476593 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [22:58:11] (03CR) 10Hashar: service-pipeline: add image tag on git tag push (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani) [22:58:12] (03CR) 10Hashar: service-pipeline: add image tag on git tag push (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/476600 (https://phabricator.wikimedia.org/T209088) (owner: 10Thcipriani)