You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
206 lines
4.9 KiB
206 lines
4.9 KiB
Metadata-Version: 2.0
|
|
Name: queuelib
|
|
Version: 1.5.0
|
|
Summary: Collection of persistent (disk-based) queues
|
|
Home-page: https://github.com/scrapy/queuelib
|
|
Author: Scrapy project
|
|
Author-email: info@scrapy.org
|
|
License: BSD
|
|
Description-Content-Type: UNKNOWN
|
|
Platform: Any
|
|
Classifier: Development Status :: 5 - Production/Stable
|
|
Classifier: License :: OSI Approved :: BSD License
|
|
Classifier: Operating System :: OS Independent
|
|
Classifier: Programming Language :: Python
|
|
Classifier: Programming Language :: Python :: 2
|
|
Classifier: Programming Language :: Python :: 2.7
|
|
Classifier: Programming Language :: Python :: 3
|
|
Classifier: Programming Language :: Python :: 3.3
|
|
Classifier: Programming Language :: Python :: 3.4
|
|
Classifier: Programming Language :: Python :: Implementation :: CPython
|
|
Classifier: Programming Language :: Python :: Implementation :: PyPy
|
|
|
|
========
|
|
queuelib
|
|
========
|
|
|
|
.. image:: https://secure.travis-ci.org/scrapy/queuelib.png?branch=master
|
|
:target: http://travis-ci.org/scrapy/queuelib
|
|
|
|
.. image:: https://img.shields.io/codecov/c/github/scrapy/queuelib/master.svg
|
|
:target: http://codecov.io/github/scrapy/queuelib?branch=master
|
|
:alt: Coverage report
|
|
|
|
|
|
Queuelib is a collection of persistent (disk-based) queues for Python.
|
|
|
|
Queuelib goals are speed and simplicity. It was originally part of the `Scrapy
|
|
framework`_ and stripped out on its own library.
|
|
|
|
Note: Queuelib isn't thread-safe.
|
|
|
|
Requirements
|
|
============
|
|
|
|
* Python 2.7 or Python 3.3
|
|
* no external library requirements
|
|
|
|
Installation
|
|
============
|
|
|
|
You can install Queuelib either via the Python Package Index (PyPI) or from
|
|
source.
|
|
|
|
To install using pip::
|
|
|
|
$ pip install queuelib
|
|
|
|
To install using easy_install::
|
|
|
|
$ easy_install queuelib
|
|
|
|
If you have downloaded a source tarball you can install it by running the
|
|
following (as root)::
|
|
|
|
# python setup.py install
|
|
|
|
FIFO/LIFO disk queues
|
|
=====================
|
|
|
|
Queuelib provides FIFO and LIFO queue implementations.
|
|
|
|
Here is an example usage of the FIFO queue::
|
|
|
|
>>> from queuelib import FifoDiskQueue
|
|
>>> q = FifoDiskQueue("queuefile")
|
|
>>> q.push(b'a')
|
|
>>> q.push(b'b')
|
|
>>> q.push(b'c')
|
|
>>> q.pop()
|
|
b'a'
|
|
>>> q.close()
|
|
>>> q = FifoDiskQueue("queuefile")
|
|
>>> q.pop()
|
|
b'b'
|
|
>>> q.pop()
|
|
b'c'
|
|
>>> q.pop()
|
|
>>>
|
|
|
|
The LIFO queue is identical (API-wise), but importing ``LifoDiskQueue``
|
|
instead.
|
|
|
|
PriorityQueue
|
|
=============
|
|
|
|
A discrete-priority queue implemented by combining multiple FIFO/LIFO queues
|
|
(one per priority).
|
|
|
|
First, select the type of queue to be used per priority (FIFO or LIFO)::
|
|
|
|
>>> from queuelib import FifoDiskQueue
|
|
>>> qfactory = lambda priority: FifoDiskQueue('queue-dir-%s' % priority)
|
|
|
|
Then instantiate the Priority Queue with it::
|
|
|
|
>>> from queuelib import PriorityQueue
|
|
>>> pq = PriorityQueue(qfactory)
|
|
|
|
And use it::
|
|
|
|
>>> pq.push(b'a', 3)
|
|
>>> pq.push(b'b', 1)
|
|
>>> pq.push(b'c', 2)
|
|
>>> pq.push(b'd', 2)
|
|
>>> pq.pop()
|
|
b'b'
|
|
>>> pq.pop()
|
|
b'c'
|
|
>>> pq.pop()
|
|
b'd'
|
|
>>> pq.pop()
|
|
b'a'
|
|
|
|
RoundRobinQueue
|
|
===============
|
|
|
|
Has nearly the same interface and implementation as a Priority Queue except
|
|
that each element must be pushed with a (mandatory) key. Popping from the
|
|
queue cycles through the keys "round robin".
|
|
|
|
Instantiate the Round Robin Queue similarly to the Priority Queue::
|
|
|
|
>>> from queuelib import RoundRobinQueue
|
|
>>> rr = RoundRobinQueue(qfactory)
|
|
|
|
And use it::
|
|
|
|
>>> rr.push(b'a', '1')
|
|
>>> rr.push(b'b', '1')
|
|
>>> rr.push(b'c', '2')
|
|
>>> rr.push(b'd', '2')
|
|
>>> rr.pop()
|
|
b'a'
|
|
>>> rr.pop()
|
|
b'c'
|
|
>>> rr.pop()
|
|
b'b'
|
|
>>> rr.pop()
|
|
b'd'
|
|
|
|
|
|
Mailing list
|
|
============
|
|
|
|
Use the `scrapy-users`_ mailing list for questions about Queuelib.
|
|
|
|
Bug tracker
|
|
===========
|
|
|
|
If you have any suggestions, bug reports or annoyances please report them to
|
|
our issue tracker at: http://github.com/scrapy/queuelib/issues/
|
|
|
|
Contributing
|
|
============
|
|
|
|
Development of Queuelib happens at GitHub: http://github.com/scrapy/queuelib
|
|
|
|
You are highly encouraged to participate in the development. If you don't like
|
|
GitHub (for some reason) you're welcome to send regular patches.
|
|
|
|
All changes require tests to be merged.
|
|
|
|
Tests
|
|
=====
|
|
|
|
Tests are located in `queuelib/tests` directory. They can be run using
|
|
`nosetests`_ with the following command::
|
|
|
|
nosetests
|
|
|
|
The output should be something like the following::
|
|
|
|
$ nosetests
|
|
.............................................................................
|
|
----------------------------------------------------------------------
|
|
Ran 77 tests in 0.145s
|
|
|
|
OK
|
|
|
|
License
|
|
=======
|
|
|
|
This software is licensed under the BSD License. See the LICENSE file in the
|
|
top distribution directory for the full license text.
|
|
|
|
Versioning
|
|
==========
|
|
|
|
This software follows `Semantic Versioning`_
|
|
|
|
.. _Scrapy framework: http://scrapy.org
|
|
.. _scrapy-users: http://groups.google.com/group/scrapy-users
|
|
.. _Semantic Versioning: http://semver.org/
|
|
.. _nosetests: https://nose.readthedocs.org/en/latest/
|
|
|
|
|