LADIES AND GENTLEMEN, may I present to you: "BLAZINGLY FAST PERFORMANCE"
Yeah so I have made some major changes regarding the optimization of the code, and let me tell you that when I say that I have managed to squeeze out a massive performance increase, I do mean it.
The test included two timed values, "TOTAL" which is the exec from start to end, and "INSERT" which is the time for the insertion of data into primula.
So here are my results of the testing:
(no optimization)
TOTAL: 74.348s
INSERT: 34.667s
(optimized, non-headless)
TOTAL: 32.492s
INSERT: 9.729s
(optimized, headless)
TOTAL: 28.956s
INSERT: 7.229s
So essentially the optimized code is about 61% faster than the original code, and while that might not sound like a massive increase to some of you, then you have got to factor in the fact that the puppeteer instance must wait for the Primula server to get every request and then respond. In other terms, you can't perform it any faster since you are limited to the server speed!
So just to put it into perspective, I am essentially operating on the limit of what is possible for the speed of the server and my network connection.
Now is there some way that I can reduce the time even further, sure but at most I think I will be able to shave off a few seconds or so, and the work simply isn't worth it.
Anyhow, archways404 singing out, peace!