directory in wordpress large number of categories and pages issue

I planned to create a business directory.

I started creating categories and when it reached around 20,000 . My website was not loading up properly.

I planned to have 80,000 categories and more than 500,000 pages. Is there any way around, that I can do that?

Waiting for some good ideas and ways to sort this problem.