Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

peeringdb sync on clean database django.core.exceptions.ValidationError: {'route_server': [u'Enter a valid URL.']} #38

Closed
Mikeburke14 opened this issue Apr 29, 2020 · 15 comments
Closed

peeringdb sync on clean database django.core.exceptions.ValidationError: {'route_server': [u'Enter a valid URL.']} #38

Mikeburke14 opened this issue Apr 29, 2020 · 15 comments
Assignees
@arnoldnipper
Labels
bug
Milestone
2 Consensus Reached

Comments

@Mikeburke14
Copy link

Hi,

Currently, the sync is failing with django.core.exceptions.ValidationError: {'route_server': [u'Enter a valid URL.']}

We normally run the following set of commands before sync to clear the database as we find ourselves hitting sync issues fairly often and this normally fixes it most of the time

 USE peering;
SET FOREIGN_KEY_CHECKS = 0; 
TRUNCATE TABLE peeringdb_ix_facility; 
TRUNCATE TABLE peeringdb_network_contact; 
TRUNCATE TABLE peeringdb_ixlan_prefix; 
TRUNCATE TABLE peeringdb_network_facility; 
TRUNCATE TABLE peeringdb_network_ixlan; 
TRUNCATE TABLE peeringdb_facility; 
TRUNCATE TABLE peeringdb_ix; 
TRUNCATE TABLE peeringdb_ixlan; 
TRUNCATE TABLE peeringdb_organization; 
TRUNCATE TABLE peeringdb_network; 
SET FOREIGN_KEY_CHECKS = 1;"

peeringdb sync

However today we are getting the following trace on sync

Syncing to https://www.peeringdb.com/api
Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
Fetching & updating all: org
Updates to be processed: 18275
Fetching & updating all: fac
Updates to be processed: 3678
Fetching & updating all: net
Updates to be processed: 18670
Traceback (most recent call last):
  File "/usr/local/bin/peeringdb", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/peeringdb/cli.py", line 62, in main
    return handler(config=cfg, **vars(options))
  File "/usr/local/lib/python2.7/dist-packages/peeringdb/commands.py", line 20, in _wrapped
    r = func(*a, **k)
  File "/usr/local/lib/python2.7/dist-packages/peeringdb/commands.py", line 219, in handle
    client.update_all(rs)
  File "/usr/local/lib/python2.7/dist-packages/peeringdb/_update.py", line 66, in update_all
    self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/usr/local/lib/python2.7/dist-packages/peeringdb/_update.py", line 78, in _atomic_update
    sync_func()
  File "/usr/local/lib/python2.7/dist-packages/peeringdb/_update.py", line 66, in <lambda>
    self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/usr/local/lib/python2.7/dist-packages/peeringdb/_tasks_sequential.py", line 77, in _wrapped
    return _consume_task(gen)
  File "/usr/local/lib/python2.7/dist-packages/peeringdb/_tasks_sequential.py", line 55, in _consume_task
    item = gen.send(r)
  File "/usr/local/lib/python2.7/dist-packages/peeringdb/_tasks_sequential.py", line 38, in gather
    for r in job:
  File "/usr/local/lib/python2.7/dist-packages/peeringdb/_update.py", line 287, in sync_row
    B.clean(obj)
  File "/usr/local/lib/python2.7/dist-packages/django_peeringdb/client_adaptor/backend.py", line 145, in clean
    obj.full_clean()
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1171, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'route_server': [u'Enter a valid URL.']}

# peeringdb --version
peeringdb 1.0.0

So it seems that someone has an invalid route server URL that is causing the sync validation to fail.
@Mikeburke14 Mikeburke14 mentioned this issue Apr 29, 2020
Closed
@netravnen
Copy link

netravnen commented Apr 30, 2020 •
edited 

$ curl -snGL https://www.peeringdb.com/api/net > $HOME/peeringdb-networks.json
$ jq '.data |
  .[] |
  select(.route_server != "") |
  [.id, .route_server]' $HOME/peeringdb-networks.json |
  jq '@csv' |
  sed 's/\\"//g;s/"//g' |
  grep '@'

22021,ssh://lg@lg.as207910.net

https://peeringdb.com/net/22021

Potentially this one that breaks it... 🤔

@Mikeburke14 Question) When did you begin experiencing the current synchronization errors? (I.e. Last successful run (timestamp) + first run when the error began (timestamp))

@Mikeburke14
Copy link
Author

@netravnen I don't have the exact timing sorry but it was first noticed Wednesday morning

@netravnen
Copy link

https://github.com/django/django/blob/master/django/core/validators.py#L63-L138

@netravnen
Copy link

I don't have the exact timing sorry but it was first noticed Wednesday morning

@Mikeburke14 ... tried fresh sync of data using MariaDB on Debian 10.3 using Python3.7... No issues detected 🤔

@Mikeburke14
Copy link
Author

Hi @netravnen

It's interesting that it works for you but not me so I did some more testing over the weekend.

First thing I did was to make sure django was up to date:

  Downloading Django-1.11.29-py2.py3-none-any.whl (6.9 MB)
     |████████████████████████████████| 6.9 MB 13.4 MB/s
Requirement already satisfied, skipping upgrade: pytz in /usr/local/lib/python2.7/dist-packages (from django) (2019.2)
Installing collected packages: django
  Attempting uninstall: django
    Found existing installation: Django 1.8.19
    Uninstalling Django-1.8.19:
      Successfully uninstalled Django-1.8.19
Successfully installed django-1.11.29

No change but I enabled debug mode which also pointed to net/22021 being the cause of the issue so I think something has changed with django itself.

So I then swapped to use python3 to see if that is any better

Requirement already up-to-date: django_peeringdb in /usr/local/lib/python3.6/dist-packages (2.0.0)
Requirement already satisfied, skipping upgrade: django-inet<0.5,>=0.3.2 in /usr/local/lib/python3.6/dist-packages (from django_peeringdb) (0.4.0)
Requirement already satisfied, skipping upgrade: django-countries>=0.1.0 in /usr/local/lib/python3.6/dist-packages (from django_peeringdb) (6.1.2)
Requirement already satisfied, skipping upgrade: twentyc.rpc<0.5,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from django_peeringdb) (0.4.0)
Requirement already satisfied, skipping upgrade: django-handleref<0.6,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from django_peeringdb) (0.5.0)
Requirement already satisfied, skipping upgrade: requests>=2.10.0 in /usr/lib/python3/dist-packages (from twentyc.rpc<0.5,>=0.4.0->django_peeringdb) (2.18.4)
Requirement already satisfied, skipping upgrade: six<2.0,>=1.10 in /usr/lib/python3/dist-packages (from twentyc.rpc<0.5,>=0.4.0->django_peeringdb) (1.11.0)

# pip3 install --upgrade django
Requirement already up-to-date: django in /usr/local/lib/python3.6/dist-packages (3.0.6)
Requirement already satisfied, skipping upgrade: asgiref~=3.2 in /usr/local/lib/python3.6/dist-packages (from django) (3.2.7)
Requirement already satisfied, skipping upgrade: sqlparse>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from django) (0.3.1)
Requirement already satisfied, skipping upgrade: pytz in /usr/local/lib/python3.6/dist-packages (from django) (2020.1)


# pip3 install --upgrade peeringdb
Requirement already up-to-date: peeringdb in /usr/local/lib/python3.6/dist-packages (1.0.0)
Requirement already satisfied, skipping upgrade: PyYAML>=3.11 in /usr/local/lib/python3.6/dist-packages (from peeringdb) (5.3.1)
Requirement already satisfied, skipping upgrade: twentyc.rpc>=0.3.4 in /usr/local/lib/python3.6/dist-packages (from peeringdb) (0.4.0)
Requirement already satisfied, skipping upgrade: munge>=0.1.5 in /usr/local/lib/python3.6/dist-packages (from peeringdb) (1.0.0)
Requirement already satisfied, skipping upgrade: cfu>=0.5.1 in /usr/local/lib/python3.6/dist-packages (from peeringdb) (1.5.0)
Requirement already satisfied, skipping upgrade: six<2.0,>=1.10 in /usr/lib/python3/dist-packages (from twentyc.rpc>=0.3.4->peeringdb) (1.11.0)
Requirement already satisfied, skipping upgrade: requests>=2.10.0 in /usr/lib/python3/dist-packages (from twentyc.rpc>=0.3.4->peeringdb) (2.18.4)
Requirement already satisfied, skipping upgrade: click>=5.1 in /usr/local/lib/python3.6/dist-packages (from munge>=0.1.5->peeringdb) (7.1.2)
Requirement already satisfied, skipping upgrade: future<1,>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from munge>=0.1.5->peeringdb) (0.18.2)



# peeringdb sync
Syncing to https://www.peeringdb.com/api
Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
Fetching & updating all: org
Updates to be processed: 0
Fetching & updating all: fac
Updates to be processed: 0
Fetching & updating all: net
Updates to be processed: 18715
/gbTraceback (most recent call last):
  File "/usr/local/bin/peeringdb", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/peeringdb/cli.py", line 62, in main
    return handler(config=cfg, **vars(options))
  File "/usr/local/lib/python3.6/dist-packages/peeringdb/commands.py", line 20, in _wrapped
    r = func(*a, **k)
  File "/usr/local/lib/python3.6/dist-packages/peeringdb/commands.py", line 219, in handle
    client.update_all(rs)
  File "/usr/local/lib/python3.6/dist-packages/peeringdb/_update.py", line 66, in update_all
    self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/usr/local/lib/python3.6/dist-packages/peeringdb/_update.py", line 78, in _atomic_update
    sync_func()
  File "/usr/local/lib/python3.6/dist-packages/peeringdb/_update.py", line 66, in <lambda>
    self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/usr/local/lib/python3.6/dist-packages/peeringdb/_tasks_async.py", line 65, in _wrapped
    return loop.run_until_complete(func(*a, **k))
  File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/dist-packages/peeringdb/_tasks_async.py", line 41, in _wrapped
    item = gen.send(r)
  File "/usr/local/lib/python3.6/dist-packages/peeringdb/_update.py", line 187, in sync_row
    obj, fetched, dangling = _sync.initialize_object(B, res, row)
  File "/usr/local/lib/python3.6/dist-packages/peeringdb/_sync.py", line 47, in initialize_object
    obj = B.get_object(B.get_concrete(res), row['id'])
  File "/usr/local/lib/python3.6/dist-packages/peeringdb/backend.py", line 26, in wrapped
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/django_peeringdb/client_adaptor/backend.py", line 97, in get_object
    return concrete.objects.get(pk=id)
  File "/usr/local/lib/python3.6/dist-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/django/db/models/query.py", line 411, in get
    num = len(clone)
  File "/usr/local/lib/python3.6/dist-packages/django/db/models/query.py", line 258, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.6/dist-packages/django/db/models/query.py", line 1261, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.6/dist-packages/django/db/models/query.py", line 57, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.6/dist-packages/django/db/models/sql/compiler.py", line 1149, in execute_sql
    cursor = self.connection.cursor()
  File "/usr/local/lib/python3.6/dist-packages/django/utils/asyncio.py", line 24, in inner
    raise SynchronousOnlyOperation(message)
django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.

This error seems pretty deep inside django You cannot call this from an async context - use a thread or sync_to_async. any ideas how to overcome it?

Thanks

@Mikeburke14
Copy link
Author

Mikeburke14 commented May 5, 2020 •
edited

Hi,

I created a simple docker file to test with the following and get the same error above

FROM python:latest

RUN pip install django
RUN pip install peeringdb
RUN pip install django_peeringdb
RUN peeringdb config show
RUN peeringdb sync

So this is using a local sqlite database and my testing was towards a MySQL server so it seems unrelated to my environment

Full Output from build

 ---> 4f7cd4269fa9
Step 2/6 : RUN pip install django
 ---> Using cache
 ---> 76786251656f
Step 3/6 : RUN pip install peeringdb
 ---> Using cache
 ---> 2114e73ef69d
Step 4/6 : RUN pip install django_peeringdb
 ---> Using cache
 ---> 566d5b472a3c
Step 5/6 : RUN peeringdb config show
 ---> Running in 713ed99ffc0d
orm:
  backend: django_peeringdb
  database:
    engine: sqlite3
    host: ''
    name: peeringdb.sqlite3
    password: ''
    port: 0
    user: ''
  migrate: true
  secret_key: ''
sync:
  only: []
  password: ''
  strip_tz: 1
  timeout: 0
  url: https://www.peeringdb.com/api
  user: ''
 ---> a37430e88d21
Removing intermediate container 713ed99ffc0d
Step 6/6 : RUN peeringdb sync
 ---> Running in 13eb44aec784
Syncing to https://www.peeringdb.com/api
Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
Fetching & updating all: org
Updates to be processed: 18323
Traceback (most recent call last):
  File "/usr/local/bin/peeringdb", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/peeringdb/cli.py", line 62, in main
    return handler(config=cfg, **vars(options))
  File "/usr/local/lib/python3.8/site-packages/peeringdb/commands.py", line 20, in _wrapped
    r = func(*a, **k)
  File "/usr/local/lib/python3.8/site-packages/peeringdb/commands.py", line 219, in handle
    client.update_all(rs)
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 66, in update_all
    self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 78, in _atomic_update
    sync_func()
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 66, in <lambda>
    self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_tasks_async.py", line 65, in _wrapped
    return loop.run_until_complete(func(*a, **k))
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_tasks_async.py", line 41, in _wrapped
    item = gen.send(r)
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 187, in sync_row
    obj, fetched, dangling = _sync.initialize_object(B, res, row)
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_sync.py", line 47, in initialize_object
    obj = B.get_object(B.get_concrete(res), row['id'])
  File "/usr/local/lib/python3.8/site-packages/peeringdb/backend.py", line 26, in wrapped
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django_peeringdb/client_adaptor/backend.py", line 97, in get_object
    return concrete.objects.get(pk=id)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 411, in get
    num = len(clone)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 258, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1261, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 57, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1149, in execute_sql
    cursor = self.connection.cursor()
  File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 24, in inner
    raise SynchronousOnlyOperation(message)
django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.
Error response from daemon: The command '/bin/sh -c peeringdb sync' returned a non-zero code: 1
Failed to deploy '<unknown> Dockerfile: Dockerfile': Can't retrieve image ID from build stream

@Mikeburke14
Copy link
Author

Hi @netravnen just wondering if you have any ideas about what we can do to solve this?

@mcmanuss8
Copy link

Seems like a bug +1 to devote some time to fix, plus some additional testing to try and catch this case

@peeringdb/pc agreed?

@shane-kerr
Copy link

+1

2 similar comments
@job
Copy link
Contributor

job commented May 12, 2020

+1

@arnoldnipper
Copy link

+1

@arnoldnipper arnoldnipper self-assigned this May 12, 2020
@arnoldnipper arnoldnipper added this to the 2 Consensus Reached milestone May 12, 2020
@bcavns01
Copy link

bcavns01 commented Jul 8, 2020

I got around this bad field by add obj.full_clean(exclude=["route_server"]) to the relevant spot in django_peeringdb/sync.py, and now it seems I'll have to do the same things for the info_traffic field:

"/usr/local/lib/python2.7/dist-packages/django_peeringdb/sync.py", line 40, in sync_obj
2020-07-08T06:03:29.582226+00:00 *** peeringdb_sync:     raise e
2020-07-08T06:03:29.582538+00:00 *** peeringdb_sync: django.core.exceptions.ValidationError: {'info_traffic': [u"Value u'10-20Tbps' is not a valid choice."]}

At this point, I feel like I should remove the raise e in sync.py to neutralize all validation so I don't have to keep chasing fields with bad data. 😞

@vegu
Copy link
Contributor

vegu commented Jul 10, 2020 •
edited

what version of django-peeringdb are you running - its not bad data, at least in this case, its validator updates that require a client upgrade to not fail validation.

'10-20Tbps' is a valid value once you upgrade to >= 2.1.0

Although it's odd that the server is letting you sync with an older version, it's supposed to block

Syncing to https://www.peeringdb.com/api
Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
Fetching & updating all: org
Your client version is incompatible with server version of the api, please install peeringdb>=0.6,<=255.0 django_peeringdb>=2.0.0.2,<=255.0

@bcavns01
Copy link

Upgraded peeringdb and django_peeringdb to latest in pip and things look ok, thanks!

@grizz
Copy link
Member

grizz commented Sep 4, 2020

Good deal, thanks! Sorry for not going through the issues before the last release to notify.

@grizz grizz closed this as completed Sep 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arnoldnipper arnoldnipper

Labels
bug
Projects
None yet
Milestone
2 Consensus Reached
Development

No branches or pull requests
9 participants
@grizz @Mikeburke14 @vegu @job @netravnen @shane-kerr @mcmanuss8 @arnoldnipper @bcavns01