[00:00:36] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:04:16] (03open) 10addbot: Bump docker-registry.wikimedia.org/wikimedia/mediawiki-libs-shellbox image group to 2024-09-11-160805 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/581 [00:04:53] (03PS1) 10Jforrester: Docker: Re-build PHP 8.1+ with php-uuid [integration/config] - 10https://gerrit.wikimedia.org/r/1072320 (https://phabricator.wikimedia.org/T373752) [00:06:06] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:37:10] 10WikimediaDebug, 10MediaWiki-Debug-Logger, 10Wikimedia-Site-requests, 07Wikimedia-production-error: Uncaught UnexpectedValueException: Udp transport "udp:///XWikimediaDebug" must specify a host - https://phabricator.wikimedia.org/T374583 (10Krinkle) 03NEW [00:37:14] 10WikimediaDebug, 10MediaWiki-Debug-Logger, 10Wikimedia-Site-requests, 07Wikimedia-production-error: Uncaught UnexpectedValueException: Udp transport "udp:///XWikimediaDebug" must specify a host - https://phabricator.wikimedia.org/T374583#10139444 (10Krinkle) Fall-out from https://gerrit.wikimedia.org/r/c/... [00:57:34] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:58:19] 10WikimediaDebug, 10MediaWiki-Debug-Logger, 10Wikimedia-Site-requests, 07Wikimedia-production-error: Uncaught UnexpectedValueException: Udp transport "udp:///XWikimediaDebug" must specify a host - https://phabricator.wikimedia.org/T374583#10139482 (10matmarex) a:03matmarex Forgot a `global $wmgUdp2logDes... [01:07:34] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [09:58:29] GitLab needs a short maintenance break at 11:30 UTC (in 90 minutes) [10:07:34] 10WikimediaDebug, 10MediaWiki-Debug-Logger, 10Wikimedia-Site-requests, 13Patch-For-Review, 07Wikimedia-production-error: Uncaught UnexpectedValueException: Udp transport "udp:///XWikimediaDebug" must specify a host - https://phabricator.wikimedia.org/T374583#10140088 (10hashar) > I wonder how difficult i... [10:35:03] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Figure out how to provision a Kubernetes cluster using Magnum and OpenTofu - https://phabricator.wikimedia.org/T372498#10140191 (10rook) >>! In T372498#10139273, @bd808 wrote: >>>! In T372498#10137248, @rook wrote: >> The hope being that beta cluster could ben... [11:20:54] FYI, with the migration of text table entries, there will be errors of "blob not found" like this https://logstash.wikimedia.org/goto/ff7d2e97a6ee78d46d70b90dad182ded for 24 hours after the run (we ran it on test2wiki yesterday). This is because of caching and automatically should resolve [11:22:53] Amir1: can you let the train deployers via this week blocking task? https://phabricator.wikimedia.org/T373641 :) [11:23:09] sounds good [11:23:15] literally copy paste your message above :D [11:23:21] thanks for the head up! [11:26:04] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T373641#10140318 (10Ladsgroup) Hi, FYI, with the migration of text table entries, there will be errors of "blob not found" like this https://logstash.wikimedi... [11:38:05] GitLab maintenance done [11:38:34] (03open) 10aklapper: Add even more temporary debug output for T356077 [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/69 (https://phabricator.wikimedia.org/T356077) [11:58:46] (03PS11) 10Arthur taylor: Add support for core's composer `phpunit:...parallel:...` commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1039669 (https://phabricator.wikimedia.org/T365976) [12:00:07] (03CR) 10Arthur taylor: Add support for core's composer `phpunit:...parallel:...` commands (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1039669 (https://phabricator.wikimedia.org/T365976) (owner: 10Arthur taylor) [12:00:59] (03CR) 10Arthur taylor: Add support for core's composer `phpunit:...parallel:...` commands (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1039669 (https://phabricator.wikimedia.org/T365976) (owner: 10Arthur taylor) [12:01:16] (03CR) 10Arthur taylor: Add support for core's composer `phpunit:...parallel:...` commands (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1039669 (https://phabricator.wikimedia.org/T365976) (owner: 10Arthur taylor) [12:17:42] (03CR) 10CI reject: [V:04-1] Add support for core's composer `phpunit:...parallel:...` commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1039669 (https://phabricator.wikimedia.org/T365976) (owner: 10Arthur taylor) [12:27:53] (03CR) 10Jforrester: [C:04-1] "I think we probably want to also have Wikimedia's PHP 7.4 binaries have uuid too, for consistency?" [integration/config] - 10https://gerrit.wikimedia.org/r/1072320 (https://phabricator.wikimedia.org/T373752) (owner: 10Jforrester) [12:38:12] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review: Tweak requests for the automated weekly Phabricator data for Tech News - https://phabricator.wikimedia.org/T373952#10140525 (10Aklapper) The last patch in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1072535 seems to... [12:41:07] 10Continuous-Integration-Config, 07Test-Infrastructure: Provide a phan test for mediawiki-config to spot errors - https://phabricator.wikimedia.org/T374615 (10Jdforrester-WMF) 03NEW [12:48:17] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10MediaWiki-Core-Tests, 10Quibble, 10Temporary accounts: Core apitests job should run extension/skin api tests as well - https://phabricator.wikimedia.org/T303255#10140607 (10JayCano) [12:54:10] 06Project-Admins: Consider archiving #sentry - https://phabricator.wikimedia.org/T374411#10140641 (10Aklapper) This may rather need the #Projects-Cleanup template if this is also about archiving the extension repositories? [12:58:41] hashar: OK to skip today? I'm trying to work out how to get request-timeout to behave better. [12:59:20] James_F: I only planned to talk about the php8.1 images and let you know I haven't had a second today to even start looking at them :) [12:59:26] then it is probably not a rush [12:59:43] but yeah happy to skip! [12:59:47] Ack. [13:32:06] 10WikimediaDebug, 10MediaWiki-Debug-Logger, 10Wikimedia-Site-requests, 07Wikimedia-production-error: Uncaught UnexpectedValueException: Udp transport "udp:///XWikimediaDebug" must specify a host - https://phabricator.wikimedia.org/T374583#10140785 (10matmarex) 05Open→03Resolved [13:41:18] 06Project-Admins, 06English-Wikipedia-New-Pages-Patrol: Sunset the phabricator tag "English-Wikipedia-New-Pages-Patrol" - https://phabricator.wikimedia.org/T363123#10140810 (10Aklapper) @TheresNoTime: Any comments / opinion, please? [13:54:43] 10Phabricator, 06collaboration-services, 10wikimedia.biterg.io: Exclude Bitergia IP from Phabricator ratelimiting - https://phabricator.wikimedia.org/T374550#10140880 (10Jelto) [13:55:35] 10Phabricator, 06collaboration-services, 10wikimedia.biterg.io: Exclude Bitergia IP from Phabricator ratelimiting - https://phabricator.wikimedia.org/T374550#10140876 (10Jelto) `34.175.7.48` belongs to Google/Google Cloud and is already included in our `request-ipblocks/cloud/gcp.yaml` list. So yes, Bitergia... [13:56:59] 10Phabricator, 06collaboration-services, 10wikimedia.biterg.io: Exclude Bitergia IP from Phabricator ratelimiting - https://phabricator.wikimedia.org/T374550#10140894 (10Aklapper) According to their feedback, that IP is static. [14:02:21] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops: Provide an mwdebug functionality on kubernetes - https://phabricator.wikimedia.org/T276994#10140924 (10jijiki) [14:02:53] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops: Provide an mwdebug functionality on kubernetes - https://phabricator.wikimedia.org/T276994#10140934 (10jijiki) [14:11:00] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops: Provide an mwdebug functionality on kubernetes - https://phabricator.wikimedia.org/T276994#10140956 (10jijiki) [14:58:01] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): buildkit v0.16.0 released - https://phabricator.wikimedia.org/T374478#10141206 (10dancy) >>! In T374478#10139303, @brennen wrote: > ` > error: failed to solve: failed to load cache key: failed to fetch remote https://github.com/conta... [15:25:54] 10Continuous-Integration-Config, 10phan, 07Test-Infrastructure: Provide a phan test for mediawiki-config to spot errors - https://phabricator.wikimedia.org/T374615#10141305 (10hashar) [15:34:17] 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Radar), 07Upstream: GitLab group permissions are not inherited by sub-groups for groups of users invited to the parent repo - https://phabricator.wikimedia.org/T300939#10141337 (10brennen) This upstream task: [[https://gitlab.com/gitlab-org/... [15:36:26] 06Project-Admins: Consider archiving #sentry - https://phabricator.wikimedia.org/T374411#10141341 (10Pppery) I at first thought it was just about archiving a Phabricator project. Then I realized there were extensions too. Then I realized the extension archival process is broken because T351543#10126970 and too f... [15:51:49] (03open) 10dancy: wmf/build.sh: Bump dockerfile-frontend to wmf-v0.15.2-1 [repos/releng/buildkit] (wmf/v0.16) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/70 (https://phabricator.wikimedia.org/T374478) [15:51:54] (03update) 10dancy: wmf/build.sh: Bump dockerfile-frontend to wmf-v0.15.2-1 [repos/releng/buildkit] (wmf/v0.16) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/70 (https://phabricator.wikimedia.org/T374478) [15:52:26] (03merge) 10dancy: wmf/build.sh: Bump dockerfile-frontend to wmf-v0.15.2-1 [repos/releng/buildkit] (wmf/v0.16) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/70 (https://phabricator.wikimedia.org/T374478) [16:38:24] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): buildkit v0.16.0 released - https://phabricator.wikimedia.org/T374478#10141687 (10dancy) I filed https://github.com/moby/buildkit/issues/5329 about the build problem. [17:04:22] 10GitLab (Infrastructure), 06collaboration-services, 07SecTeam-Processed, 07Security, 07Vuln-VulnComponent: GitLab Critical Patch Release: 17.3.2, 17.2.5, 17.1.7 - https://phabricator.wikimedia.org/T374588#10141877 (10sbassett) [17:04:51] 10GitLab (Infrastructure), 06collaboration-services, 07SecTeam-Processed, 07Security, 07Vuln-VulnComponent: GitLab Critical Patch Release: 17.3.2, 17.2.5, 17.1.7 - https://phabricator.wikimedia.org/T374588#10141878 (10sbassett) [17:51:32] (03open) 10dduvall: jobs: Define a job that publishes a single version MediaWiki image [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/79 (https://phabricator.wikimedia.org/T369115) [17:51:32] (03update) 10dduvall: jobs: Define a job that publishes a single version MediaWiki image [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/79 (https://phabricator.wikimedia.org/T369115) [17:51:44] (03update) 10dduvall: jobs: Define a job that publishes a single version MediaWiki image [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/79 (https://phabricator.wikimedia.org/T369115) [17:51:46] (03update) 10dduvall: jobs: Define a job that publishes a single version MediaWiki image [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/79 (https://phabricator.wikimedia.org/T369115) [17:52:50] (03update) 10dduvall: jobs: Define a job that publishes a single version MediaWiki image [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/79 (https://phabricator.wikimedia.org/T369115) [17:56:55] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T373641#10142092 (10dduvall) >>! In T373641#10140318, @Ladsgroup wrote: > Hi, FYI, with the migration of text table entries, there will be errors of "blob not... [18:19:52] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10observability: Explore monitoring for the GitLab runners k8s cluster - https://phabricator.wikimedia.org/T363919#10142129 (10lmata) Hi @thcipriani, I've checked with the team, and we're familiar with federation but are unsure abo... [18:44:26] 10Release-Engineering-Team (Radar), 10MediaWiki-Debug-Logger, 10observability, 07Developer Productivity, and 2 others: Consider enabling all MW log channels by default for WMF - https://phabricator.wikimedia.org/T228838#10142185 (10matmarex) It looks like the only new errors on the beta cluster appeared on... [18:56:09] (03open) 10dancy: sync_common(): Exclude /scap/ directory from rsync [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/434 [18:56:11] (03update) 10dancy: sync_common(): Exclude /scap/ directory from rsync [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/434 [18:57:37] (03update) 10dancy: sync_common(): Exclude /scap/ directory from rsync [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/434 [18:57:42] (03update) 10dancy: sync_common(): Exclude /scap/ directory from rsync [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/434 [18:58:40] (03update) 10dancy: sync_common(): Exclude /scap/ directory from rsync [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/434 [19:06:14] 10Continuous-Integration-Infrastructure, 06serviceops, 10MediaWiki-Platform-Team (Radar): Prepare WMF PHP 8.1 packages for Bullseye - https://phabricator.wikimedia.org/T372507#10142205 (10Scott_French) [19:09:03] 10Continuous-Integration-Infrastructure, 06serviceops, 13Patch-For-Review: Prepare PHP 8.1 production images - https://phabricator.wikimedia.org/T372602#10142203 (10Scott_French) 05Open→03Resolved At this point, I believe we're good to move forward with initial testing using these images, and there a... [19:10:15] (03merge) 10thcipriani: sync_common(): Exclude /scap/ directory from rsync [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/434 (owner: 10dancy) [19:10:39] 10Continuous-Integration-Infrastructure, 06serviceops, 10MediaWiki-Platform-Team (Radar): Prepare WMF PHP 8.1 packages for Bullseye - https://phabricator.wikimedia.org/T372507#10142206 (10Scott_French) 05Open→03Resolved Many thanks for driving the CI updates, @Jdforrester-WMF. I believe this wraps ev... [19:14:14] (03update) 10dancy: Initial spiderpig jobrunner implementation and data model [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/433 [19:55:07] (03update) 10dancy: Initial spiderpig jobrunner implementation and data model [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/433 [20:04:58] (03update) 10dancy: Initial spiderpig jobrunner implementation and data model [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/433 [20:54:41] (03update) 10dancy: Initial spiderpig jobrunner implementation and data model [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/433 [20:55:21] (03update) 10dancy: Initial spiderpig jobrunner implementation and data model [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/433 [21:37:25] 10Phabricator, 06collaboration-services, 10wikimedia.biterg.io: Exclude Bitergia IP from Phabricator ratelimiting - https://phabricator.wikimedia.org/T374550#10142554 (10Dzahn) There was this idea to add a list of friendly networks in our network data.yaml https://gerrit.wikimedia.org/r/c/operations/puppet/... [21:40:51] 10Phabricator, 06collaboration-services, 10wikimedia.biterg.io: Exclude Bitergia IP from Phabricator ratelimiting - https://phabricator.wikimedia.org/T374550#10142576 (10Dzahn) How urgent is this? Is this a daily problem or just like once a month? I will look more tomorrow either way. [21:41:51] 10Continuous-Integration-Infrastructure, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Requesting access to `contint-admins`, `contint-docker`, LDAP `ciadmin` for 'Arthur taylor' - https://phabricator.wikimedia.org/T373969#10142588 (10thcipriani) >>! In T373969#10130953, @Ladsgroup wrote: > This we... [22:05:40] (03update) 10dduvall: jobs: Define a job that publishes a single version MediaWiki image [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/79 (https://phabricator.wikimedia.org/T369115) [22:12:17] (03update) 10dduvall: jobs: Define a job that publishes a single version MediaWiki image [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/79 (https://phabricator.wikimedia.org/T369115) [22:27:18] (03approved) 10dancy: jobs: Define a job that publishes a single version MediaWiki image [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/79 (https://phabricator.wikimedia.org/T369115) (owner: 10dduvall) [22:28:11] (03merge) 10dduvall: jobs: Define a job that publishes a single version MediaWiki image [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/79 (https://phabricator.wikimedia.org/T369115) [22:32:41] PROBLEM - jenkins_service_running on releases1003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [22:34:41] RECOVERY - jenkins_service_running on releases1003 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [22:54:04] (03open) 10dduvall: jobs: Fix use of relativeTargetDirectory [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/80 (https://phabricator.wikimedia.org/T369115) [22:54:05] (03update) 10dduvall: jobs: Fix use of relativeTargetDirectory [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/80 (https://phabricator.wikimedia.org/T369115) [22:54:17] (03update) 10dduvall: jobs: Fix use of relativeTargetDirectory [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/80 (https://phabricator.wikimedia.org/T369115) [22:54:19] (03update) 10dduvall: jobs: Fix use of relativeTargetDirectory [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/80 (https://phabricator.wikimedia.org/T369115) [22:56:22] dancy: ^ fix for jenkins-deploy [22:56:58] (03approved) 10dancy: jobs: Fix use of relativeTargetDirectory [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/80 (https://phabricator.wikimedia.org/T369115) (owner: 10dduvall) [22:57:12] (03merge) 10dduvall: jobs: Fix use of relativeTargetDirectory [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/80 (https://phabricator.wikimedia.org/T369115) [22:58:27] ty! [22:58:57] not a fan of how these job definitions can take down the entire jenkins but i suppose if i were testing these changes in a local scap-dev environment that would not be a problem [23:01:24] !log deployed https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/79 and https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/80 to releases-jenkins [23:01:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:04:15] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T373641#10142848 (10dduvall) 05Open→03Resolved [23:16:30] (03update) 10dancy: Initial spiderpig jobrunner implementation and data model [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/433 [23:18:48] (03update) 10dancy: Initial spiderpig jobrunner implementation and data model [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/433 [23:40:47] 10Phabricator, 06collaboration-services, 10wikimedia.biterg.io: Exclude Bitergia IP from Phabricator ratelimiting - https://phabricator.wikimedia.org/T374550#10142903 (10thcipriani) >>! In T374550#10142576, @Dzahn wrote: > How urgent is this? Is this a daily problem or just like once a month? I will look mo...