View Issue Details

IDProjectCategoryView StatusLast Update
0000403AlmaLinux-8-OTHERpublic2023-06-16 08:54
ReporterHitamashi Assigned To 
PrioritynormalSeveritytextReproducibilityalways
Status newResolutionopen 
Platformx86_64OSAlmaLinux 
Summary0000403: Sync AlmaLinux 8 Appstream Repo "https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/" fails
DescriptionI'm using katello. When syncing AppStream repo https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/, it will following error "PLP0000: Importer indicated a failed response".

Additional InformationCurrent version:
pulp-server-2.21.5-1.el7.noarch
katello-3.18.4-1.el7.noarch
foreman-2.3.5-1.el7.noarch

Katello output:
{"pulp_tasks"=>
  [{"exception"=>nil,
    "task_type"=>"pulp.server.managers.repo.sync.sync",
    "_href"=>"/pulp/api/v2/tasks/d18166c2-c017-4650-8e2b-613385fa4fc2/",
    "task_id"=>"d18166c2-c017-4650-8e2b-613385fa4fc2",
    "tags"=>
     ["pulp:repository:1d58d5fa-c886-444f-af67-e046f262f20a",
      "pulp:action:sync"],
    "finish_time"=>"2023-06-07T10:30:04Z",
    "_ns"=>"task_status",
    "start_time"=>"2023-06-07T10:28:23Z",
    "traceback"=>
     "Traceback (most recent call last):\n" +
     " File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n" +
     " R = retval = fun(*args, **kwargs)\n" +
     " File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 688, in __call__\n" +
     " return super(Task, self).__call__(*args, **kwargs)\n" +
     " File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 110, in __call__\n" +
     " return super(PulpTask, self).__call__(*args, **kwargs)\n" +
     " File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n" +
     " return self.run(*args, **kwargs)\n" +
     " File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 860, in sync\n" +
     " raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))\n" +
     "PulpExecutionException: Importer indicated a failed response\n",
    "spawned_tasks"=>[],
    "progress_report"=>
     {"yum_importer"=>
       {"content"=>
         {"items_total"=>0,
          "state"=>"FINISHED",
          "error_details"=>[],
          "details"=>
           {"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0},
          "size_total"=>0,
          "size_left"=>0,
          "items_left"=>0},
        "comps"=>{"state"=>"NOT_STARTED"},
        "purge_duplicates"=>{"state"=>"NOT_STARTED"},
        "distribution"=>
         {"items_total"=>0,
          "state"=>"FINISHED",
          "error_details"=>[],
          "items_left"=>0},
        "modules"=>
         {"state"=>"FAILED",
          "error"=>
           "encoder expected a mapping type but got: '\\x1d\\x00\\x00\\x00\\x041.0\\x00\\x13\\x00\\x00\\x00\\x020\\x00\\x07\\x00\\x00\\x00common\\x00\\x00\\x00'"},
        "errata"=>{"state"=>"NOT_STARTED"},
        "metadata"=>{"state"=>"FINISHED"}}},
    "queue"=>"abcxyz",
    "state"=>"error",
    "worker_name"=>"abcxyz",
    "result"=>nil,
    "error"=>
     {"code"=>"PLP0000",
      "data"=>{},
      "description"=>"Importer indicated a failed response",
      "sub_errors"=>[]},
    "_id"=>{"$oid"=>"64805bc72fe340c8f5f4c528"},
    "id"=>"64805bc72fe340c8f5f4c528"}],
 "contents_changed"=>true,
 "poll_attempts"=>{"total"=>27, "failed"=>1}}
Tagsalmalinux8, appstream, repo
abrt_hash
URL

Activities

Hitamashi

2023-06-08 08:09

reporter   ~0000908

Another similar error

{"pulp_tasks"=>
  [{"exception"=>nil,
    "task_type"=>"pulp.server.managers.repo.sync.sync",
    "_href"=>"/pulp/api/v2/tasks/59a814b5-6332-428e-89d5-02e4ce7d403f/",
    "task_id"=>"59a814b5-6332-428e-89d5-02e4ce7d403f",
    "tags"=>
     ["pulp:repository:1d58d5fa-c886-444f-af67-e046f262f20a",
      "pulp:action:sync"],
    "finish_time"=>"2023-06-07T22:01:55Z",
    "_ns"=>"task_status",
    "start_time"=>"2023-06-07T22:00:04Z",
    "traceback"=>
     "Traceback (most recent call last):\n" +
     " File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n" +
     " R = retval = fun(*args, **kwargs)\n" +
     " File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 688, in __call__\n" +
     " return super(Task, self).__call__(*args, **kwargs)\n" +
     " File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 110, in __call__\n" +
     " return super(PulpTask, self).__call__(*args, **kwargs)\n" +
     " File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n" +
     " return self.run(*args, **kwargs)\n" +
     " File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 860, in sync\n" +
     " raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))\n" +
     "PulpExecutionException: Importer indicated a failed response\n",
    "spawned_tasks"=>[],
    "progress_report"=>
     {"yum_importer"=>
       {"content"=>
         {"items_total"=>0,
          "state"=>"FINISHED",
          "error_details"=>[],
          "details"=>
           {"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0},
          "size_total"=>0,
          "size_left"=>0,
          "items_left"=>0},
        "comps"=>{"state"=>"NOT_STARTED"},
        "purge_duplicates"=>{"state"=>"NOT_STARTED"},
        "distribution"=>
         {"items_total"=>0,
          "state"=>"FINISHED",
          "error_details"=>[],
          "items_left"=>0},
        "modules"=>
         {"state"=>"FAILED",
          "error"=>
           "strings in documents must be valid UTF-8: '\\x82\\x00\\x00\\x00\\x041.16\\x00\\x13\\x00\\x00\\x00\\x020\\x00\\x07\\x00\\x00\\x00common\\x00\\x00\\x041.22\\x00\\x13\\x00\\x00\\x00\\x020\\x00\\x07\\x00\\x00\\x00common\\x00\\x00\\x041.14\\x00\\x13\\x00\\x00\\x00\\x020\\x00\\x07\\x00\\x00\\x00common\\x00\\x00\\x041.20\\x00\\x13\\x00\\x00\\x00\\x020\\x00\\x07\\x00\\x00\\x00common\\x00\\x00\\x041.18\\x00\\x13\\x00\\x00\\x00\\x020\\x00\\x07\\x00\\x00\\x00common\\x00\\x00\\x00'"},
        "errata"=>{"state"=>"NOT_STARTED"},
        "metadata"=>{"state"=>"FINISHED"}}},
    "queue"=>"abcxyz",
    "state"=>"error",
    "worker_name"=>"abcxyz",
    "result"=>nil,
    "error"=>
     {"code"=>"PLP0000",
      "data"=>{},
      "description"=>"Importer indicated a failed response",
      "sub_errors"=>[]},
    "_id"=>{"$oid"=>"6480fde3d0953052c495338e"},
    "id"=>"6480fde3d0953052c495338e"}],
 "contents_changed"=>true,
 "poll_attempts"=>{"total"=>28, "failed"=>1}}

ccto32

2023-06-11 00:23

reporter   ~0000909

Dear AlmaLinux support,

Yes, we encountered the similar problem.

Please help.

--------

[root@xxxxxx ~]# cat /etc/redhat-release
AlmaLinux release 8.7 (Stone Smilodon)
[root@xxxxxx ~]#
[root@xxxxxx ~]# yum clean all
0 files removed
[root@xxxxxx ~]#
[root@xxxxxx ~]# yum check-update
AlmaLinux 8 - BaseOS 1.9 MB/s | 2.8 MB 00:01
AlmaLinux 8 - AppStream 166 B/s | 744 B 00:04
Errors during downloading metadata for repository 'appstream':
  - Status code: 404 for http://mirrors.nju.edu.cn/almalinux/8.8/AppStream/x86_64/os/repodata/5e704ef80796fd5fadc41eea302860a38195f8ff5ec1810f9a84e9e222aba28b-comps-AppStream.x86_64.xml (IP: 210.28.130.3)
  - Status code: 404 for http://mirrors.neusoft.edu.cn/almalinux/8.8/AppStream/x86_64/os/repodata/48fac7b41b32ca8d9a67f4f9ca8496ac973e34275260a96c497738d5c394c122-updateinfo.xml.gz (IP: 219.216.128.25)
  - Status code: 404 for https://mirror.kku.ac.th/almalinux/8.8/AppStream/x86_64/os/repodata/894203b4259b93c5d07945b3c6dfc0d051f5f8be146126d6b9638a3a8b3f6ee1-primary.xml.gz (IP: 202.28.95.174)
  - Status code: 404 for http://mirror.01link.hk/almalinux/8.8/AppStream/x86_64/os/repodata/48fac7b41b32ca8d9a67f4f9ca8496ac973e34275260a96c497738d5c394c122-updateinfo.xml.gz (IP: 101.78.134.82)
...
  - Status code: 404 for http://hkg.mirror.rackspace.com/almalinux/8.8/AppStream/x86_64/os/repodata/faaf5696360c1e09891fbd9790c0afe156ebdbae83e6ae5bf7bc61c2671a0f7b-modules.yaml.gz (IP: 180.150.156.88)
Error: Failed to download metadata for repo 'appstream': Yum repo downloading error: Downloading error(s): repodata/894203b4259b93c5d07945b3c6dfc0d051f5f8be146126d6b9638a3a8b3f6ee1-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/7c0abdd4c0c956c1999dbfcfbb7a0f9866d5ab2899f51daaa884db4ff4607e4d-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/5e704ef80796fd5fadc41eea302860a38195f8ff5ec1810f9a84e9e222aba28b-comps-AppStream.x86_64.xml - Cannot download, all mirrors were already tried without success; repodata/faaf5696360c1e09891fbd9790c0afe156ebdbae83e6ae5bf7bc61c2671a0f7b-modules.yaml.gz - Cannot download, all mirrors were already tried without success; repodata/48fac7b41b32ca8d9a67f4f9ca8496ac973e34275260a96c497738d5c394c122-updateinfo.xml.gz - Cannot download, all mirrors were already tried without success
[root@xxxxxx ~]#

alukoshko

2023-06-12 13:50

administrator   ~0000910

Hello. Is it possible that you use some caching proxy and it has outdated repomd.xml?
There is no such file:
http://mirrors.neusoft.edu.cn/almalinux/8.8/AppStream/x86_64/os/repodata/48fac7b41b32ca8d9a67f4f9ca8496ac973e34275260a96c497738d5c394c122-updateinfo.xml.gz
In repomd.xml on the same mirror:
http://mirrors.neusoft.edu.cn/almalinux/8.8/AppStream/x86_64/os/repodata/repomd.xml

alukoshko

2023-06-12 13:50

administrator   ~0000911

Hitamashi, is this still reproducible?
When it happened for the first time?

Hitamashi

2023-06-14 04:01

reporter   ~0000912

Hello alukoshko,
The issue is still happening on my side. As I last check, it started from 18 or 19 May
FYI, we don't use any proxy and we are using the official repo: https://repo.almalinux.org (tried some other mirrors too but it is the same)

Katello seems to get the correct metadata files but it cannot parse it. Here are some more logs on my katello server

2023-06-13T22:00:11.638443+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [77e20cb0] Downloading metadata from https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/.
2023-06-13T22:00:12.030816+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/repomd.xml.
2023-06-13T22:00:12.644387+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [77e20cb0] Downloading metadata from https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/.
2023-06-13T22:00:13.037398+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/repomd.xml.
2023-06-13T22:00:13.650485+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [77e20cb0] Downloading metadata from https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/.
2023-06-13T22:00:14.039833+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/repomd.xml.
2023-06-13T22:00:14.659084+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [77e20cb0] Downloading metadata from https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/.
2023-06-13T22:00:15.056043+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/repomd.xml.
2023-06-13T22:00:17.217328+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/c506a63e6ec2310bbe4700a2cea294829a84a115bf3b2d27d74d040595619c87-comps.xml.
2023-06-13T22:00:17.240123+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/3abb7e78188af4e9bdb9237a1285e44015f444635fea14a2e4c6269508f92d63-updateinfo.xml.gz.
2023-06-13T22:00:17.328541+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/25e7f52dcf83e5d7de9e6ca2e8f2b346731ea65b9cc9382510e96a75628f401c-modules.yaml.
2023-06-13T22:00:17.715379+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/7e6c2e6eaa89ac622558caec1942f4946c3e28e41c9e6409911be897bcb46619-filelists.xml.gz.
2023-06-13T22:00:17.745900+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/909ab6d80107e46aba7fa94265f63b7a41320e651b2982e1cae3690360182726-primary.xml.gz.
2023-06-13T22:00:18.268155+00:00 katello-server pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/repodata/c8b0b2d955b55f1c74db148ae6e7b9643f9d93e4d178e8dae889f526778f7e20-other.xml.gz.
2023-06-13T22:01:59.257174+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) strings in documents must be valid UTF-8: '\x82\x00\x00\x00\x041.16\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.22\x00\x13\x00\x00\
x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.14\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.20\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.18\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x00'
2023-06-13T22:01:59.257370+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) Traceback (most recent call last):
2023-06-13T22:01:59.257550+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 312, in run
2023-06-13T22:01:59.257762+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) repair=self.validate)
2023-06-13T22:01:59.257971+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/modularity.py", line 415, in synchronize
2023-06-13T22:01:59.258153+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) remainder = add_defaults(repository, defaults, repair=repair)
2023-06-13T22:01:59.258312+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/modularity.py", line 340, in add_defaults
2023-06-13T22:01:59.258470+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) add_default(repository, default, model)
2023-06-13T22:01:59.258624+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/modularity.py", line 287, in add_default
2023-06-13T22:01:59.258775+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) model.save_and_import_content(path)
2023-06-13T22:01:59.258945+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py", line 936, in save_and_import_content
2023-06-13T22:01:59.259099+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) self.save()
2023-06-13T22:01:59.259252+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 324, in save
2023-06-13T22:01:59.259414+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) object_id = collection.save(doc, **write_concern)
2023-06-13T22:01:59.259566+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 2180, in save
2023-06-13T22:01:59.259717+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) check_keys, False, manipulate, write_concern)
2023-06-13T22:01:59.259894+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 709, in _update
2023-06-13T22:01:59.260058+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) codec_options=self.codec_options).copy()
2023-06-13T22:01:59.260213+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 216, in command
2023-06-13T22:01:59.260363+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) self._raise_connection_failure(error)
2023-06-13T22:01:59.260515+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 343, in _raise_connection_failure
2023-06-13T22:01:59.260665+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) raise error
2023-06-13T22:01:59.260841+00:00 katello-server pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [77e20cb0] (32148-18944) InvalidStringData: strings in documents must be valid UTF-8: '\x82\x00\x00\x00\x041.16\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.22\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.14\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.20\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x041.18\x00\x13\x00\x00\x00\x020\x00\x07\x00\x00\x00common\x00\x00\x00'

alukoshko

2023-06-15 16:59

administrator   ~0000916

> 18 or 19 May

May 18 was release of 8.8. So it should be related.

Hitamashi

2023-06-16 07:43

reporter   ~0000917

```
> 18 or 19 May
May 18 was release of 8.8. So it should be related.
```
Do you have any update on the way to generate the metadata when moving to 8.8?
FYI, it works fine with 8.7 AppStream: https://repo.almalinux.org/almalinux/8.7/AppStream/x86_64/os/

Hitamashi

2023-06-16 08:54

reporter   ~0000918

It works now with the following workaround: https://forums.developer.nvidia.com/t/error-syncing-rhel8-cuda-repo/176276/9
Similar bugs on satellite: https://bugzilla.redhat.com/show_bug.cgi?id=1920511

I tried this last week but no result but now it works somehow. I think this is more about katello/pulp issue

Issue History

Date Modified Username Field Change
2023-06-08 03:47 Hitamashi New Issue
2023-06-08 08:09 Hitamashi Note Added: 0000908
2023-06-11 00:23 ccto32 Note Added: 0000909
2023-06-12 13:50 alukoshko Note Added: 0000910
2023-06-12 13:50 alukoshko Note Added: 0000911
2023-06-14 04:01 Hitamashi Note Added: 0000912
2023-06-15 10:20 Hitamashi Tag Attached: repo
2023-06-15 10:20 Hitamashi Tag Attached: almalinux8
2023-06-15 10:20 Hitamashi Tag Attached: appstream
2023-06-15 16:59 alukoshko Note Added: 0000916
2023-06-16 07:43 Hitamashi Note Added: 0000917
2023-06-16 08:54 Hitamashi Note Added: 0000918