See this excellent article on how to improve page speed when using Google fonts: https://www.sitepoint.com/improve-page-performance-font-loader/
When your site is browsed by a visitor whose device does not have the Lato and Raleway fonts installed (the majority) the fonts are loaded from Google. This will delay the load of your pages (even as zips these fonts add 1MB to the data that needs to be sent to the users browser + of course 2*2 more response requests).
As stated in the article, Theme speed could be optimized by initially specifying a web-safe font (one that is “built in” to most devices) in CSS and using JS to switch once the Google fonts have downloaded.
I doubt I’ll have time to try it out but I’d be interested in the results if anyone else does.