Flat file DB guidance

IndigoIdentity said:
You have clients that are each invoking a request to some external API and the server is dealing with each of these requests. You can force the client to make the request to the Google API if that is the case, can you not?

Take a look here too: https://codelabs.developers.google.com/codelabs/lovefield/index.html?index=../../index#0

Thing is, i'm unsure of how this would work if it even does when it comes to multiple users accessing the data, I figure it to be something more like a cookie but I may be wrong? It still seems like it would bring the benefits in terms of offline data storage and also in that link it's dealing with monitoring stock data so is applicable...

However, without creating a client side application and working with what you have got, you'd still need to cache the data on the server side...

So, instead of the website querying the Google API directly for each request, implement a cache for the results so that when your server gets 10 clients all making the same query at once, it will fetch and store the data once then realise that the other 9 are the exact same thing so will return the cached result instead of querying the Google API again?

Where does it store that data? Who cares, sqlite is fine if its just a small thing but dont query the Google API 100 times is my point... I think that is the problem that you're facing.

That sounds to me like a scalable solution, you could have many users making use of this and its not going to get slower and slower each time someone else joins in?
Click to expand...

Close, look if I could use a DB it would have been a walk in a park this is what I have in my offline setup

Cronjob:

PHP: 
//Stock Watch

// set quotes 
$quotes = array("CLI", "CFR", "VOD", "DSY", "CCO", "MFL", "SHP", "TAS", "MTN", "SYG", "SGL", "SNV");

foreach ($quotes as $quote) {
    $url = "http://www.bloomberg.com/markets/api/quote-page/$quote:SJ";
    $json = file_get_contents($url);
    $json = json_decode($json);

    $company = $json->basicQuote->name;
    $price = $json->basicQuote->price;
    $percentage = round($percentage = $json->basicQuote->percentChange1Day, 2, PHP_ROUND_HALF_UP);
    $modified_time = date("Y-m-d H:i:s");

    
    // Update the quotes in the Database
    $sql = "UPDATE stock_history SET price = :price, percentage = :percentage, time = :time WHERE company = :company";
    $statement = $db->prepare($sql);
    $statement->execute(array(':company' => $company, ':price' => $price, ':percentage' => $percentage, ':time' => $modified_time));

}


//Market News RSS Feed

$rss = new DOMDocument();
$rss->load('http://feeds.24.com/articles/Fin24/Markets/rss');
$feed = array();
foreach ($rss->getElementsByTagName('item') as $node) {
    $item = array ( 
        'title' => $node->getElementsByTagName('title')->item(0)->nodeValue,
        'desc' => $node->getElementsByTagName('description')->item(0)->nodeValue,
        'link' => $node->getElementsByTagName('link')->item(0)->nodeValue,
        'date' => $node->getElementsByTagName('pubDate')->item(0)->nodeValue,
        );
    array_push($feed, $item);
}
$limit = 5;
for($x=0;$x<$limit;$x++) {
    $title = str_replace(' & ', ' &amp; ', $feed[$x]['title']);

    $pattern = '(Fin24\\.com)';
    $replacement = '';

    $a = preg_replace($pattern, $replacement, $title);

    $re1='.*?';	# Non-greedy match on filler
    $re2='(\\|)';	# Any Single Character 1
    $replacement_2 = '';

    $title = preg_replace("/".$re1.$re2."/is", $replacement_2, $a);
    $link = $feed[$x]['link'];
    $description = $feed[$x]['desc'];
    $date = date('l F d, Y', strtotime($feed[$x]['date']));
    $unique_hash = hash('SHA256', $title.$link);
  
$statement = $db->prepare("INSERT IGNORE INTO market_feed (`title`, `description`, `link`, `date`, `unique_hash` ) VALUES (:title, :description, :link, :date, :unique_hash)");
$statement->execute(array(':title' => $title, ':description' => $description, ':link' => $link, ':date' => $date, ':unique_hash' => $unique_hash));

}

Then whenever a user hits the site I merely read back the data:

PHP: 
<?php  
//Select Stocks to show

$sql = "SELECT company, price, percentage FROM stock_history";

$statement = $db->prepare($sql);
$statement->execute();

//Display the stocks

while ($row = $statement->fetch(PDO::FETCH_ASSOC)) {
    echo "<div class=\"stock"; if ($row["percentage"] >= 0) {echo ' green';} else { echo ' red';} echo "\" id=\"stock-1\">
    <span class=\"symbol\">"; echo $row["company"];
    echo "</span>
    <span class=\"change\">";if ($row["percentage"] > 0) {echo '+';} else { echo '';} ;
    echo $row["percentage"];echo "%</span>";
    echo "<br>";
    echo "<span class=\"price\">"; echo $row["price"]; echo " cents</span>";
    echo "</div>";
  }

?>


The above is what I want to implement into PW, but the forum software is quite a headache to work with very convoluted, but getting off topic.

Point is I want to do what I did there, but without a database.
 
Thor187 said:
The above is what I want to implement into PW, but the forum software is quite a headache to work with very convoluted, but getting off topic.

Point is I want to do what I did there, but without a database.
Click to expand...

Fair enough, that would work fairly well on the back end but regarding without a database, it's why i suggested having the client query for the data and use something like love field to store it / make it easier to work with then your server does none of the heavy work, each client does its own and the server doesn't actually need a db at all.

Can you motivate for me in lamens terms why you'd not like to use a database here? If there any one reason which makes you think that it's not a good choice? Or just for fun this is?
 
How often are you querying the data source?


If your file being served is actual.html then:

Why don't you
- query the data source.
- On success, write an HTML file you want the user to view but with filename aaa.html
- On success of the aaa.html file creation, mv the aaa.html to actual.html
- Put a short caching timeout on actual.html and let your webserver handle the caching?

Kind of like what you have in the first post but leave out dumping the data to a file and rad it from there, just create the html file and let visitors read it directly.

Edit:

The process above would be done via cronjob
 
Last edited:
Also trying to picture the end result here... So you load a page it shows current values for some data. Then what? Isn't it supposed to be like realtime polling the server to check if these values are changed?
 
rward said:
How often are you querying the data source?


If your file being served is actual.html then:

Why don't you
- query the data source.
- On success, write an HTML file you want the user to view but with filename aaa.html
- On success of the aaa.html file creation, mv the aaa.html to actual.html
- Put a short caching timeout on actual.html and let your webserver handle the caching?

Kind of like what you have in the first post but leave out dumping the data to a file and rad it from there, just create the html file and let visitors read it directly.

Edit:

The process above would be done via cronjob
Click to expand...
I like this!

This I'm going to do anyway now even if I don't use it as I feel this is something I'll probably come back to and use for something else.

Thank you.
 
@rward you always have the interesting ideas, made me think of a twist to yours.

What if its not writing an .html document but a .json file and the page uses ajax to query the file to populate its values?
 
I skimmed through this as there is quite a bit, so forgive me if off the mark...

Have you not considered web sockets? I use this a lot for multiple clients that render the same data. Client registers with the server, then sits and waits for the server notification of data change.
This has reduced the load on our servers in a big way, compared to when we did polling.
 
DominionZA said:
I skimmed through this as there is quite a bit, so forgive me if off the mark...

Have you not considered web sockets? I use this a lot for multiple clients that render the same data. Client registers with the server, then sits and waits for the server notification of data change.
This has reduced the load on our servers in a big way, compared to when we did polling.
Click to expand...
Definitely something I'll be reading up on tonight.

I'm aware of it, just have not actually implemented it so will give it a study.
 
this isn't perfect, but at least it gives you something to think about, especially around separation

PHP: 
//internal methods and error handling omitted for brevity

public function processRequest($stock) {
    return getStockValue($stock);
}

private function getStockValue($stock) {
    $result = getStockValueFromCache($stock);  //redis, filecache, apc, memcached, etc
    if ($result == null) {
        try {
            $result = getStockValueFromApi($stock); //call API, deserialize response
            saveStockValueToCache($stock, $value); //redis, filecache, apc, memcached, etc
            saveStockValueToStorage($stock, $value); //mysql, sqlite file, csv file, etc
        } catch ($exception) {
            $result = getStockValueFromStorage($stock); //mysql, sqlite file, csv file, etc
            saveStockValueToCache($stock, $value); //redis, filecache, apc, memcached, etc
        }
    }
    return $result;
}
 
DominionZA said:
I skimmed through this as there is quite a bit, so forgive me if off the mark...

Have you not considered web sockets? I use this a lot for multiple clients that render the same data. Client registers with the server, then sits and waits for the server notification of data change.
This has reduced the load on our servers in a big way, compared to when we did polling.
Click to expand...
As long as he uses a library that falls back to polling. Web sockets aren't supported in every browser and firewalls tend to render them useless.
 
Hamster said:
FTFY
Click to expand...

Oh, I agree. You want your company's production solutions to share as many components as possible.

For test projects I'm all for trying anything you want, but the thing is that you shouldn't aim immediately for the lowest common denominator (in this case a file), try some of the more modern (but well tested) things (such as for example Redis) first. They've already done all the hard work for you by figuring out how to reliably serialize a lot of data into a file.
 
IndigoIdentity said:
@rward you always have the interesting ideas, made me think of a twist to yours.

What if its not writing an .html document but a .json file and the page uses ajax to query the file to populate its values?
Click to expand...

That is a way to do it but that's back to almost what Thor first posted.
On each view you have 1 more file read that needs to happen, unless the json query can also be cached?
 
animal531 said:
Oh, I agree. You want your company's production solutions to share as many components as possible.

For test projects I'm all for trying anything you want, but the thing is that you shouldn't aim immediately for the lowest common denominator (in this case a file), try some of the more modern (but well tested) things (such as for example Redis) first. They've already done all the hard work for you by figuring out how to reliably serialize a lot of data into a file.
Click to expand...

Hamster gets herpes when it comes to Redis, we've been through numerous discussions about it. ;)


Code: 
https://github.com/mrjgreen/cacher

$fileBackend = new Cacher\Backends\File('path/to/tmpstorage');

// Uses any compatible redis library. EG nrk/predis, irediscent/irediscent
$redisBackend = new Cacher\Backends\Redis(new Predis\Client($config));

$stackedCache = new Cacher($redisBackend, new Cacher($fileBackend));

// Looks in redis then falls back to file before calling the callback function
$stackedCache->get('key', function(){
  return 'value';
});

There done.
 
I get the feeling you guys are over engineering this .. :)
 
This is one of my arrays:

PHP: 
array (size=1)
  0 => 
    array (size=16)
      'id' => string '888357114685955' (length=15)
      't' => string 'DBXWD' (length=5)
      'e' => string 'JSE' (length=3)
      'l' => string '2,426.00' (length=8)
      'l_fix' => string '2426.00' (length=7)
      'l_cur' => string 'ZAC2,426.00' (length=11)
      's' => string '0' (length=1)
      'ltt' => string '3:29PM GMT+2' (length=12)
      'lt' => string 'Dec 12, 3:29PM GMT+2' (length=20)
      'lt_dts' => string '2016-12-12T15:29:11Z' (length=20)
      'c' => string '+17.00' (length=6)
      'c_fix' => string '17.00' (length=5)
      'cp' => string '0.71' (length=4)
      'cp_fix' => string '0.71' (length=4)
      'ccol' => string 'chg' (length=3)
      'pcls_fix' => string '2409' (length=4)

If I want ZAC2,426.00 to display as R 2 426 I assume I will have to do preg_replace correct?
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X