Skip to content
This repository was archived by the owner on Dec 17, 2019. It is now read-only.

Conversation

@bunelr
Copy link

@bunelr bunelr commented Aug 29, 2014

Merged the tip of the original repo into our branch, and resolved the merge conflicts so that our code doesn't have to be changed, but that makes it different from the main repo because they put some backwards incompatible changes.
It made the switch from urllib2 to provide the connection to requests (so urllib3 inside).

On top of that, I changed the connection to pingdom api to use a request.Session instead of simple requests request which allows to cache the connection and give a small speed boost.

From a profile run from our crawler, that spends most of the time doing the http calls:

Before:

build/bdist.linux-x86_64/egg/pingdom/connection.py:87(fetch) ->    516    0.006  495.193  /usr/lib/python2.7/urllib2.py:122(urlopen

After:

build/bdist.linux-x86_64/egg/pingdom/connection.py:71(fetch) ->    520    0.010  370.924  /home/vagrant/dogweb/python/local/lib/python2.7/site-packages/requests-2.2.0-py2.7.egg/requests/sessions.py:387(get)

516/520 is the number of times the http cales are made. 495/370 is the cumulative time spent doing them.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants