I have a graphite setup using a docker , now i setup another stand-alone graphite instance , i setup them as a cluster meaning both of them can see data each other. The weird thing is that the docker instance can see data from stand alone instance. But stand alone instance can’t see data in docker instance. i doubt it’s a network issue, but it’s not , i can see stand-alone sending query to docker. i don’t see any error in the graphite web log, i finally find a way to turn on the log. We need to turn on DEBUG=True in local_settings.py , then edit file .$GRAPHITE_ROOT/webapp/graphite/remote_storage.py, in the FindRequest class change the line “suppressErrors = True” to “suppressErrors = False”.
Then at about line 94 , you need to change this:
except Exception as e:
self.store.fail()
if not self.suppressErrors:
raise e
else:
results = []
Restart web server, you will see some error like this when using the Inspector
Django Version: 1.6.11.7
Exception Type: ValueError
Exception Value:
unsupported pickle protocol: 4
Exception Location: /opt/graphite/webapp/graphite/remote_storage.py in get_results, line 86
Python Executable: /usr/bin/python
Python Version: 2.7.5
Python Path:
['/opt/graphite/webapp',
'/usr/lib64/python27.zip',
'/usr/lib64/python2.7',
'/usr/lib64/python2.7/plat-linux2',
'/usr/lib64/python2.7/lib-tk',
'/usr/lib64/python2.7/lib-old',
'/usr/lib64/python2.7/lib-dynload',
'/usr/lib64/python2.7/site-packages',
'/usr/lib64/python2.7/site-packages/gtk-2.0',
'/usr/lib/python2.7/site-packages',
'/opt/graphite/webapp',
'/opt/graphite/webapp/graphite/thirdparty']
Server time: Sat, 8 Feb 2020 02:51:48 +0000
You will see an error : unsupported pickle protocol: 4 . , it turns out that the docker instance is using python3 while my standalone is using version 2.7.5. python3 have many protocol number for pickle , it can go from 0 to 5 , while python 2 can go only from 0-3
My fixes is to upgrade my python to python3