PHP speed question

[Other Pineapple Smurf]

That's what I currently used it for hehehe.


In the loop I am removing some pipes and funny character from a string begotten from the api loop runs for 300 currency items lel.

Clearly not efficient so I'll be "optimizing tonight"

Quicker to throw more resources at the problem (RAM, CPU, disk) than to optimise. Increase the timeout if you have to and to improve user experience flush progress output if you want.

Trust me, 300 is peanuts. Worry about it when it hits 1 million. I always say cross that bridge when you get there as I see developers spend too much time trying to optimise that they end up writing voodoo code that is impossible to debug or expand. I see them waste weeks on code that never sees the volumes they say it will.

The best way to handle long running processes is to do them as an async service or as a cron job.

 

[[)roi(]]

Each loop irreteration takes 384ms

So 300 x 384ms

Then the regex takes 0.025ms

So it's 300 x (384 + 0.025)


I can't do anything on the API speed but I can use something else instead of regular expression.

So that's a loop within a loop scenario; Are you sure your figures are correct? (300 x 384ms = 115seconds, approximately 2 minutes??)

 

[Thor]

Quicker to throw more resources at the problem (RAM, CPU, disk) than to optimise. Increase the timeout if you have to and to improve user experience flush progress output if you want.

Trust me, 300 is peanuts. Worry about it when it hits 1 million. I always say cross that bridge when you get there as I see developers spend too much time trying to optimise that they end up writing voodoo code that is impossible to debug or expand. I see them waste weeks on code that never sees the volumes they say it will.

The best way to handle long running processes is to do them as an async service or as a cron job.

I'm having it as a cron job yes. Just thought can't hurt to get my best practices in. If regex is bad then I eliminate them.

Best practice is more important to me.

 

[Thor]

So that's a loop within a loop scenario; Are you sure your figures are correct? (300 x 384ms = 115seconds, approximately 2 minutes??)

No no not a loop within a loop altho there are some for each loops.

But yea it takes about 2 minutes to get all the currencies hence why I run it as a cron job and writes it to the dB like you guys suggested and then users see the data from the db thus page loads are 0.021ms

 

[biometrics]

I'm having it as a cron job yes. Just thought can't hurt to get my best practices in. If regex is bad then I eliminate them.

Best practice is more important to me.

"Best practice." Define it.

 

[[)roi(]]

No no not a loop within a loop altho there are some for each loops.

But yea it takes about 2 minutes to get all the currencies hence why I run it as a cron job and writes it to the dB like you guys suggested and then users see the data from the db thus page loads are 0.021ms

2 minutes for a background job sounds perfect, unless you're running a stock portal, once an hour currency updates should suffice? Anyway with 0.021ms page load you shouldn't even bother?

FYI Typically the only way to optimise multiple sweep scenarios, ... is to write your own tokenizer and parser (or to use something a Parser Combinator Framework), that alone should tell you it's more bother than benefit.

 

[Other Pineapple Smurf]

"Best practice." Define it.

Code that works and has unit testing. Thats best practice.

 

[DA-LION-619]

Code that works and has unit testing. Thats best practice.

I think he wanted a definition from Thor.

 

[Other Pineapple Smurf]

I'm having it as a cron job yes. Just thought can't hurt to get my best practices in. If regex is bad then I eliminate them.

Best practice is more important to me.

If you are using a cron job then make sure you use lock files to prevent multiple instances of the same cron script running. Even if you only intend to run it once every hour or day, you might end up wanting to run it every 5 minutes in the future.

 

[Thor]

2 minutes for a background job sounds perfect, unless you're running a stock portal, once an hour currency updates should suffice? Anyway with 0.021ms page load you shouldn't even bother?

FYI Typically the only way to optimise multiple sweep scenarios, ... is to write your own tokenizer and parser (or to use something a Parser Combinator Framework), that alone should tell you it's more bother than benefit.

no no I am saying after doing what you guys suggested the page load is now 0.021 before it was minutes hehe.

and yea it's a stock portal of sorts so I have the cron job to run every 8 minutes as it takes up to 4 minutes ( 2 on average ) and the stocks update every 15 on the api so I have some leniency there.

It works 100% so I thought now is probably a good time to get rid of bad practises like regex

 

[Thor]

I think he wanted a definition from Thor.

Here you are again, without substance and never with advice. Makes one wonder.

 

[DA-LION-619]

Here you are again, without substance and never with advice. Makes one wonder.

You never take the advice.
And this isn't the first thread nor the last thread. You choose to cherry pick what you need till it doesn't work out then make a new thread.

 

[biometrics]

You never take the advice.
And this isn't the first thread nor the last thread. You choose to cherry pick what you need till it doesn't work out then make a new thread.

He gets an A for enthorsiasm though.

 

[Thor]

You never take the advice.
And this isn't the first thread nor the last thread. You choose to cherry pick what you need till it doesn't work out then make a new thread.

Complete BS. but whatever floats your boat son.

 

[semaphore]

Code that works and has unit testing. Thats best practice.

That hardly constitutes as best practise.

 

[[)roi(]]

no no I am saying after doing what you guys suggested the page load is now 0.021 before it was minutes hehe.

and yea it's a stock portal of sorts so I have the cron job to run every 8 minutes as it takes up to 4 minutes ( 2 on average ) and the stocks update every 15 on the api so I have some leniency there.

It works 100% so I thought now is probably a good time to get rid of bad practises like regex

Regex isn't a bad practice by itself; as with anything it's how you use it that makes it bad or good (With Great Power Comes Great Responsibility). Plus don't be scared to make mistakes, that's the way you learn + you're always going to cringe a little when you look at old code.

Re your cron jobs, heed this advice:

If you are using a cron job then make sure you use lock files to prevent multiple instances of the same cron script running. Even if you only intend to run it once every hour or day, you might end up wanting to run it every 5 minutes in the future.

 

[Thor]

Regex isn't a bad practice by itself; as with anything it's how you use it that makes it bad or good (With Great Power Comes Great Responsibility). Plus don't be scared to make mistakes, that's the way you learn + you're always going to cringe a little when you look at old code.

Re your cron jobs, heed this advice:

I will need some guidance here.

My cronjob is one file inside that file it has 4 loops

Stock

ETFs

Currency

News


It writes to 4 tables:

stocks
etfs
currency_rates
news

That is it.


So I think what is confusing to me is this part:

multiple instances of the same cron script

Since to me this only makes sense if I have 4 cron jobs using the same file ie file.txt.

 

[biometrics]

So I think what is confusing to me is this part:

Cron #1 can start Script #1 and while it's running Cron #2 could fire and run Script #1. Unless your code can cater for it it can cause ***. Rather abort Cron #2 while Cron #1 is still running. There are various ways of doing it but a quick and dirty way is simply to create a file while Cron #1 is running and deleting it when it's done. Check for the existence of this file in Cron #2.

 

[[)roi(]]

Here's a PHP blog on this to help you get started.
http://abhinavsingh.com/how-to-use-locks-in-php-cron-jobs-to-avoid-cron-overlaps/

 

[Thor]

Cron #1 can start Script #1 and while it's running Cron #2 could fire and run Script #1. Unless your code can cater for it it can cause ***. Rather abort Cron #2 while Cron #1 is still running. There are various ways of doing it but a quick and dirty way is simply to create a file while Cron #1 is running and deleting it when it's done. Check for the existence of this file in Cron #2.

Here's a PHP blog on this to help you get started.
http://abhinavsingh.com/how-to-use-locks-in-php-cron-jobs-to-avoid-cron-overlaps/

Yea yea that is what i thought, it does not apply to my right now, but I am fully aware of this.

I will use Solo when I get to that stage. I will not be pagerdutied 2am when my server reaches a load of 90 because I have 3365 instances of the script running.