There has been some recent discussion at Reddit of an attempt I made to turn some data from arXiv.0rg into a visualization of the relationships between research areas in mathematics. I was never completely pleased with the dataset.
When searching for papers, one gets this result:
Your query resulted in too many hits, only 1000 hits are being displayed. These are not necessarily the 1000 most recent papers. We recommend that you try a more specific search.
This limitation on the number of results makes it difficult to get more data. I wrote the arXiv.0rg a few times (I am sure they are quite busy) to request access to more data but I was never able to get a reply.
If anybody has an ideas for other databases, I would be glad to hear about it.