Collectors
LensDev uses collectors to retrieve and normalize data from external sources.
A Collector is a specialized module in LensDev responsible for interacting with a specific API or data source. Each collector handles its own authentication, rate limiting, and schema mapping to ensure the data is ready for the LensDev pipeline.
🌐
stable Wikipedia
General knowledge and context.
💻
stable GitHub
Technical repositories and code.
📄
stable arXiv
Scientific pre-prints and papers.
📚
beta OpenAlex
Global research ecosystem data.
🆔
beta Crossref
Metadata for millions of DOIs.
🌍
beta GDELT
Global news and events database.
Extending Collectors
Don't see the source you need? LensDev is designed to be extensible. You can build your own collector in Python by implementing our standard interface.
Read the Contribution Guide