0000403: Sync AlmaLinux 8 Appstream Repo "https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/" fails - MantisBT

ID	Project	Category	View Status	Date Submitted	Last Update

0000403	AlmaLinux-8	-OTHER	public	2023-06-08 03:47	2023-06-16 08:54

Reporter	Hitamashi	Assigned To
Priority	normal	Severity	text	Reproducibility	always
Status	new	Resolution	open
Platform	x86_64	OS	AlmaLinux

Summary	0000403: Sync AlmaLinux 8 Appstream Repo "https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/" fails
Description	I'm using katello. When syncing AppStream repo https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/, it will following error "PLP0000: Importer indicated a failed response".
Additional Information	Current version: pulp-server-2.21.5-1.el7.noarch katello-3.18.4-1.el7.noarch foreman-2.3.5-1.el7.noarch Katello output: {"pulp_tasks"=> [{"exception"=>nil, "task_type"=>"pulp.server.managers.repo.sync.sync", "_href"=>"/pulp/api/v2/tasks/d18166c2-c017-4650-8e2b-613385fa4fc2/", "task_id"=>"d18166c2-c017-4650-8e2b-613385fa4fc2", "tags"=> ["pulp:repository:1d58d5fa-c886-444f-af67-e046f262f20a", "pulp:action:sync"], "finish_time"=>"2023-06-07T10:30:04Z", "_ns"=>"task_status", "start_time"=>"2023-06-07T10:28:23Z", "traceback"=> "Traceback (most recent call last):\n" + " File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n" + " R = retval = fun(args, kwargs)\n" + " File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 688, in __call__\n" + " return super(Task, self).__call__(args, *kwargs)\n" + " File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 110, in __call__\n" + " return super(PulpTask, self).__call__(args, *kwargs)\n" + " File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n" + " return self.run(args, **kwargs)\n" + " File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 860, in sync\n" + " raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))\n" + "PulpExecutionException: Importer indicated a failed response\n", "spawned_tasks"=>[], "progress_report"=> {"yum_importer"=> {"content"=> {"items_total"=>0, "state"=>"FINISHED", "error_details"=>[], "details"=> {"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0}, "size_total"=>0, "size_left"=>0, "items_left"=>0}, "comps"=>{"state"=>"NOT_STARTED"}, "purge_duplicates"=>{"state"=>"NOT_STARTED"}, "distribution"=> {"items_total"=>0, "state"=>"FINISHED", "error_details"=>[], "items_left"=>0}, "modules"=> {"state"=>"FAILED", "error"=> "encoder expected a mapping type but got: '\\x1d\\x00\\x00\\x00\\x041.0\\x00\\x13\\x00\\x00\\x00\\x020\\x00\\x07\\x00\\x00\\x00common\\x00\\x00\\x00'"}, "errata"=>{"state"=>"NOT_STARTED"}, "metadata"=>{"state"=>"FINISHED"}}}, "queue"=>"abcxyz", "state"=>"error", "worker_name"=>"abcxyz", "result"=>nil, "error"=> {"code"=>"PLP0000", "data"=>{}, "description"=>"Importer indicated a failed response", "sub_errors"=>[]}, "_id"=>{"$oid"=>"64805bc72fe340c8f5f4c528"}, "id"=>"64805bc72fe340c8f5f4c528"}], "contents_changed"=>true, "poll_attempts"=>{"total"=>27, "failed"=>1}}
Tags	almalinux8, appstream, repo

abrt_hash
URL

Hitamashi 2023-06-08 08:09 reporter ~0000908	Another similar error {"pulp_tasks"=> [{"exception"=>nil, "task_type"=>"pulp.server.managers.repo.sync.sync", "_href"=>"/pulp/api/v2/tasks/59a814b5-6332-428e-89d5-02e4ce7d403f/", "task_id"=>"59a814b5-6332-428e-89d5-02e4ce7d403f", "tags"=> ["pulp:repository:1d58d5fa-c886-444f-af67-e046f262f20a", "pulp:action:sync"], "finish_time"=>"2023-06-07T22:01:55Z", "_ns"=>"task_status", "start_time"=>"2023-06-07T22:00:04Z", "traceback"=> "Traceback (most recent call last):\n" + " File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n" + " R = retval = fun(args, kwargs)\n" + " File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 688, in __call__\n" + " return super(Task, self).__call__(args, *kwargs)\n" + " File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 110, in __call__\n" + " return super(PulpTask, self).__call__(args, *kwargs)\n" + " File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n" + " return self.run(args, **kwargs)\n" + " File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 860, in sync\n" + " raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))\n" + "PulpExecutionException: Importer indicated a failed response\n", "spawned_tasks"=>[], "progress_report"=> {"yum_importer"=> {"content"=> {"items_total"=>0, "state"=>"FINISHED", "error_details"=>[], "details"=> {"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0}, "size_total"=>0, "size_left"=>0, "items_left"=>0}, "comps"=>{"state"=>"NOT_STARTED"}, "purge_duplicates"=>{"state"=>"NOT_STARTED"}, "distribution"=> {"items_total"=>0, "state"=>"FINISHED", "error_details"=>[], "items_left"=>0}, "modules"=> {"state"=>"FAILED", "error"=> "strings in documents must be valid UTF-8: '\\x82\\x00\\x00\\x00\\x041.16\\x00\\x13\\x00\\x00\\x00\\x020\\x00\\x07\\x00\\x00\\x00common\\x00\\x00\\x041.22\\x00\\x13\\x00\\x00\\x00\\x020\\x00\\x07\\x00\\x00\\x00common\\x00\\x00\\x041.14\\x00\\x13\\x00\\x00\\x00\\x020\\x00\\x07\\x00\\x00\\x00common\\x00\\x00\\x041.20\\x00\\x13\\x00\\x00\\x00\\x020\\x00\\x07\\x00\\x00\\x00common\\x00\\x00\\x041.18\\x00\\x13\\x00\\x00\\x00\\x020\\x00\\x07\\x00\\x00\\x00common\\x00\\x00\\x00'"}, "errata"=>{"state"=>"NOT_STARTED"}, "metadata"=>{"state"=>"FINISHED"}}}, "queue"=>"abcxyz", "state"=>"error", "worker_name"=>"abcxyz", "result"=>nil, "error"=> {"code"=>"PLP0000", "data"=>{}, "description"=>"Importer indicated a failed response", "sub_errors"=>[]}, "_id"=>{"$oid"=>"6480fde3d0953052c495338e"}, "id"=>"6480fde3d0953052c495338e"}], "contents_changed"=>true, "poll_attempts"=>{"total"=>28, "failed"=>1}}

ccto32 2023-06-11 00:23 reporter ~0000909	Dear AlmaLinux support, Yes, we encountered the similar problem. Please help. -------- [root@xxxxxx ~]# cat /etc/redhat-release AlmaLinux release 8.7 (Stone Smilodon) [root@xxxxxx ~]# [root@xxxxxx ~]# yum clean all 0 files removed [root@xxxxxx ~]# [root@xxxxxx ~]# yum check-update AlmaLinux 8 - BaseOS 1.9 MB/s \| 2.8 MB 00:01 AlmaLinux 8 - AppStream 166 B/s \| 744 B 00:04 Errors during downloading metadata for repository 'appstream': - Status code: 404 for http://mirrors.nju.edu.cn/almalinux/8.8/AppStream/x86_64/os/repodata/5e704ef80796fd5fadc41eea302860a38195f8ff5ec1810f9a84e9e222aba28b-comps-AppStream.x86_64.xml (IP: 210.28.130.3) - Status code: 404 for http://mirrors.neusoft.edu.cn/almalinux/8.8/AppStream/x86_64/os/repodata/48fac7b41b32ca8d9a67f4f9ca8496ac973e34275260a96c497738d5c394c122-updateinfo.xml.gz (IP: 219.216.128.25) - Status code: 404 for https://mirror.kku.ac.th/almalinux/8.8/AppStream/x86_64/os/repodata/894203b4259b93c5d07945b3c6dfc0d051f5f8be146126d6b9638a3a8b3f6ee1-primary.xml.gz (IP: 202.28.95.174) - Status code: 404 for http://mirror.01link.hk/almalinux/8.8/AppStream/x86_64/os/repodata/48fac7b41b32ca8d9a67f4f9ca8496ac973e34275260a96c497738d5c394c122-updateinfo.xml.gz (IP: 101.78.134.82) ... - Status code: 404 for http://hkg.mirror.rackspace.com/almalinux/8.8/AppStream/x86_64/os/repodata/faaf5696360c1e09891fbd9790c0afe156ebdbae83e6ae5bf7bc61c2671a0f7b-modules.yaml.gz (IP: 180.150.156.88) Error: Failed to download metadata for repo 'appstream': Yum repo downloading error: Downloading error(s): repodata/894203b4259b93c5d07945b3c6dfc0d051f5f8be146126d6b9638a3a8b3f6ee1-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/7c0abdd4c0c956c1999dbfcfbb7a0f9866d5ab2899f51daaa884db4ff4607e4d-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/5e704ef80796fd5fadc41eea302860a38195f8ff5ec1810f9a84e9e222aba28b-comps-AppStream.x86_64.xml - Cannot download, all mirrors were already tried without success; repodata/faaf5696360c1e09891fbd9790c0afe156ebdbae83e6ae5bf7bc61c2671a0f7b-modules.yaml.gz - Cannot download, all mirrors were already tried without success; repodata/48fac7b41b32ca8d9a67f4f9ca8496ac973e34275260a96c497738d5c394c122-updateinfo.xml.gz - Cannot download, all mirrors were already tried without success [root@xxxxxx ~]#

alukoshko 2023-06-12 13:50 administrator ~0000910	Hello. Is it possible that you use some caching proxy and it has outdated repomd.xml? There is no such file: http://mirrors.neusoft.edu.cn/almalinux/8.8/AppStream/x86_64/os/repodata/48fac7b41b32ca8d9a67f4f9ca8496ac973e34275260a96c497738d5c394c122-updateinfo.xml.gz In repomd.xml on the same mirror: http://mirrors.neusoft.edu.cn/almalinux/8.8/AppStream/x86_64/os/repodata/repomd.xml

alukoshko 2023-06-12 13:50 administrator ~0000911	Hitamashi, is this still reproducible? When it happened for the first time?

Hitamashi 2023-06-14 04:01 reporter ~0000912	Hello alukoshko, The issue is still happening on my side. As I last check, it started from 18 or 19 May FYI, we don't use any proxy and we are using the official repo: https://repo.almalinux.org (tried some other mirrors too but it is the same) Katello seems to get the correct metadata files but it cannot parse it. Here are some more logs on my katello server 2023-06-13T22:00:11.638443+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [77e20cb0] Downloading metadata from https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/. 2023-06-13T22:00:12.030816+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/repomd.xml. 2023-06-13T22:00:12.644387+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [77e20cb0] Downloading metadata from https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/. 2023-06-13T22:00:13.037398+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/repomd.xml. 2023-06-13T22:00:13.650485+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [77e20cb0] Downloading metadata from https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/. 2023-06-13T22:00:14.039833+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/repomd.xml. 2023-06-13T22:00:14.659084+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [77e20cb0] Downloading metadata from https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/. 2023-06-13T22:00:15.056043+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/repomd.xml. 2023-06-13T22:00:17.217328+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/c506a63e6ec2310bbe4700a2cea294829a84a115bf3b2d27d74d040595619c87-comps.xml. 2023-06-13T22:00:17.240123+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/3abb7e78188af4e9bdb9237a1285e44015f444635fea14a2e4c6269508f92d63-updateinfo.xml.gz. 2023-06-13T22:00:17.328541+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/25e7f52dcf83e5d7de9e6ca2e8f2b346731ea65b9cc9382510e96a75628f401c-modules.yaml. 2023-06-13T22:00:17.715379+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/7e6c2e6eaa89ac622558caec1942f4946c3e28e41c9e6409911be897bcb46619-filelists.xml.gz. 2023-06-13T22:00:17.745900+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/909ab6d80107e46aba7fa94265f63b7a41320e651b2982e1cae3690360182726-primary.xml.gz. 2023-06-13T22:00:18.268155+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/c8b0b2d955b55f1c74db148ae6e7b9643f9d93e4d178e8dae889f526778f7e20-other.xml.gz. 2023-06-13T22:01:59.257174+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) strings in documents must be valid UTF-8: '\x82\x00\x00\x00\x041.16\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.22\x00\x13\x00\x00\ x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.14\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.20\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.18\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x00' 2023-06-13T22:01:59.257370+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) Traceback (most recent call last): 2023-06-13T22:01:59.257550+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 312, in run 2023-06-13T22:01:59.257762+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) repair=self.validate) 2023-06-13T22:01:59.257971+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/modularity.py", line 415, in synchronize 2023-06-13T22:01:59.258153+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) remainder = add_defaults(repository, defaults, repair=repair) 2023-06-13T22:01:59.258312+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/modularity.py", line 340, in add_defaults 2023-06-13T22:01:59.258470+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) add_default(repository, default, model) 2023-06-13T22:01:59.258624+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/modularity.py", line 287, in add_default 2023-06-13T22:01:59.258775+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) model.save_and_import_content(path) 2023-06-13T22:01:59.258945+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py", line 936, in save_and_import_content 2023-06-13T22:01:59.259099+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) self.save() 2023-06-13T22:01:59.259252+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 324, in save 2023-06-13T22:01:59.259414+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) object_id = collection.save(doc, **write_concern) 2023-06-13T22:01:59.259566+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 2180, in save 2023-06-13T22:01:59.259717+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) check_keys, False, manipulate, write_concern) 2023-06-13T22:01:59.259894+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 709, in _update 2023-06-13T22:01:59.260058+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) codec_options=self.codec_options).copy() 2023-06-13T22:01:59.260213+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 216, in command 2023-06-13T22:01:59.260363+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) self._raise_connection_failure(error) 2023-06-13T22:01:59.260515+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 343, in _raise_connection_failure 2023-06-13T22:01:59.260665+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) raise error 2023-06-13T22:01:59.260841+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) InvalidStringData: strings in documents must be valid UTF-8: '\x82\x00\x00\x00\x041.16\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.22\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.14\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.20\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.18\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x00'

alukoshko 2023-06-15 16:59 administrator ~0000916	> 18 or 19 May May 18 was release of 8.8. So it should be related.

Hitamashi 2023-06-16 07:43 reporter ~0000917	``` > 18 or 19 May May 18 was release of 8.8. So it should be related. ``` Do you have any update on the way to generate the metadata when moving to 8.8? FYI, it works fine with 8.7 AppStream: https://repo.almalinux.org/almalinux/8.7/AppStream/x86_64/os/

Hitamashi 2023-06-16 08:54 reporter ~0000918	It works now with the following workaround: https://forums.developer.nvidia.com/t/error-syncing-rhel8-cuda-repo/176276/9 Similar bugs on satellite: https://bugzilla.redhat.com/show_bug.cgi?id=1920511 I tried this last week but no result but now it works somehow. I think this is more about katello/pulp issue

Date Modified	Username	Field	Change
2023-06-08 03:47	Hitamashi	New Issue
2023-06-08 08:09	Hitamashi	Note Added: 0000908
2023-06-11 00:23	ccto32	Note Added: 0000909
2023-06-12 13:50	alukoshko	Note Added: 0000910
2023-06-12 13:50	alukoshko	Note Added: 0000911
2023-06-14 04:01	Hitamashi	Note Added: 0000912
2023-06-15 10:20	Hitamashi	Tag Attached: repo
2023-06-15 10:20	Hitamashi	Tag Attached: almalinux8
2023-06-15 10:20	Hitamashi	Tag Attached: appstream
2023-06-15 16:59	alukoshko	Note Added: 0000916
2023-06-16 07:43	Hitamashi	Note Added: 0000917
2023-06-16 08:54	Hitamashi	Note Added: 0000918