Realtime updates from MySQL

SBSP

Senior Member
Joined
Sep 7, 2007
Messages
663
I have a project that I'm working and I need a real time update from the database. It's currently running on my own PC using WAMP but I will soon run this on a hosted server replacing the current solution.

The Web fronted (PHP) has a bootstrap toggle switch. I'm using Ajax with the toggle switch to change the value in a MySQL database between 1 and 0.

Then I have an Micro Controller (ESP8266) that checks the saved value in the database from a remote location on a very regular basis. If the server returns 1 its on and if 0 it's off. Ideally I want this to be real time and have it check the status every second or less.

The front end allows you to add as many devices as you want, also this service allows other users to connect to it and they may also add as many devices as they like. So say I have 1000 users and each has 5 devices or more connected.

That's 5000 devices checking the database constantly. I feel this is a bad idea. Also sounds like something that could be detected by the hosting provider as a DDOS attack ? I don't nearly have that many users at the moment and devices are checking the DB every 5 Seconds. Currently the setup behaves like a bridge. It fetches multiple statuses then distributes multiple statuses to all the devices via wire locally from the bridge which is a raspberry PI or Arduino with an Ethernet Shield. But I want to get rid of the bridge and have wireless devices connect directly to the service.

Am I under minding MySQL or is it up to this? Or Should I be using something else instead of MySQL? Or Should I approach the whole scenario differently ? Maybe using some sort publish and subscribe I.E RSS (I have no idea how that works yet)

Quickly searching goolge I found this https://redis.io/ But its a server that needs to be installed. Seeing this is a free service and not generating any income I really don't want to pay for a Linux VPS.

Any advice ? Also bear in mind the ESP8266 client has basic HTTP functionality, It's not very efficient when it comes to processing data, So using something like Node.js is going to be tricky if not impossible. Its CPU is tiny and does not have a heck of allot of memory Its not meant to crunch through code neither does it have a built in Java run time environment so everything has to be done on the hosted server.
 
Last edited:

Pho3nix

The Legend
Joined
Jul 31, 2009
Messages
30,589
Might be a dumb question but why a switch? What's it's actual purpose? To prevent real time updates?
 

SBSP

Senior Member
Joined
Sep 7, 2007
Messages
663

DA-LION-619

Honorary Master
Joined
Aug 22, 2009
Messages
13,777
Is this data relational? Why use a SQL db?

Is it important? If not use Redis or caching of your web framework.
 
Last edited:

SBSP

Senior Member
Joined
Sep 7, 2007
Messages
663
Is this data relational? Why use a SQL db?

Is it important? If not use Redis or caching of your web framework.

It's not relational. The database table holds the info about who the device belongs to , a Secret key, the device name, date last updated, status, date created and a few other things. So yes its somewhat important. Are you familiar with with redis ? is it a framework and can it be used with any hosting cpanel based hosting provider ?



Reddit used a CDN for /r/place , maybe this approach may work for you to?

https://redditblog.com/2017/04/13/how-we-built-rplace/

Very Interesting, Looks like something that I could use and even if I cant , Its a nice read!
 

SBSP

Senior Member
Joined
Sep 7, 2007
Messages
663
I think I have it! :)

Start off with a session variable, if its empty get the the switch status, Date and which ever fields I need from the database and then Pack it in to the variable ? Are session variables stored in server memory or as a file ? (Think i read somewhere in a file which might not be desirable)

Un-Pack data from session variable when the wireless devices request it.

Update the database less frequently , Don't know how to go about this part yet, the question i when ? Say after 1000 requests or maybe when the session ends. Sounds like it could go out of sync if the wireless client left unexpectedly. (The device might be on but the database shows its off)
 

Thor

Honorary Master
Joined
Jun 5, 2014
Messages
44,236
What about socket.io then your clients ping the API.

Comet or WebSockets as well
 
Last edited:

gkm

Expert Member
Joined
May 10, 2005
Messages
1,519
Unless you are using a fairly beefy MySQL, it is not going to survive 5000tps from your 5000 devices doing one request per second each.

From what you are trying to do, sounds more like something in the realm of MQTT (https://www.google.com/search?q=mqtt+ESP8266), rather than some high volume distributed polling. MQTT will probably not even wake up for this kind of load, since it only sends data on a state change. You can probably publish your 0/1 as a retained message on a channel for the device to keep things simple.

AWS IOT also has a free tier and might be a match for what you are doing: https://www.google.com/search?q=aws+iot+ESP8266 Then you can maybe use the device shadow to indicate if your lamp should be on or off, or something like that. I have limited knowledge in this area, but it sounds like the kind of thing you are doing.
 

Solitude

Executive Member
Joined
Jul 23, 2008
Messages
7,312
Why does it need to query the database constantly though? Can't you keep a map of the devices in memory that you update and check? If a value changes then it changes both in the database and in memory.

I'm not sure how the micro controller accesses the database but if you can make both the site and the database access work from the same program then keeping everything in memory will be much more efficient.
 

rward

Senior Member
Joined
Oct 26, 2007
Messages
865
If you can support websockets then do like the reddit people did and go CDN and websockets.
We decided to cache at the CDN (Fastly) layer because it was simple to implement and it meant the cache was as close to clients as possible which would help response speed. Requests for the full state of the board were cached by Fastly with an expiration of 1 second. We also added the stale-while-revalidate cache control header option to prevent more requests from falling through than we wanted when the cached board expired. Fastly maintains around 33 POPs which do independent caching, so we expected to get at most 33 requests per second for the full board.

We used our websocket service to publish updates to all the clients. We’ve had success using it in production for reddit live threads with over 100,000 simultaneous viewers, live PM notifications, and other features. The websocket service has also been a cornerstone of our past April Fools projects such as The Button and Robin. For r/place, clients maintained a websocket connection to receive real-time tile placement updates.
 

SBSP

Senior Member
Joined
Sep 7, 2007
Messages
663
Why does it need to query the database constantly though? Can't you keep a map of the devices in memory that you update and check? If a value changes then it changes both in the database and in memory.

I'm not sure how the micro controller accesses the database but if you can make both the site and the database access work from the same program then keeping everything in memory will be much more efficient.

The micro controller literally sends 2 values using a Form Post $_POST['EMAIL'] and $_POST['SECRET'] then waits for a 1 or a 0 if the web server returns anything else it will ignore it.

Looks like i have it sorted now.
Exactly what I have done. I grab the username and secret key from $_POST['EMAIL'] and $_POST['SECRET'] , then check if the Key and Username exist in the a session variable that contains a multi-directional array. If not then clean the array, query the database and repopulate the array with the list of available devices (Off coarse only devices belonging to the username will be be populated into the array) then check the if the Username and key exist again. If not then fail otherwise return 1 or 0 ,

One drawback! But luckily it work is my favor. If user wasn't found in the current session array after checking the database and the session variable was re-populated with the array its not immediately available until the page was reloaded. Fortunately it fails the login but will successfully login when the page is reloaded again on the next attempt and not the other way around. For now I can live with it.

Thanks rward! The idea came from your redit post.

So now the load is on the web server and PHP only. the database will be only be queried when the login fails or when the session ended or when you added a new device that does not yet exist in the memory.

I might have to revisit this again look at what Thor187 suggested by using web sockets. I found some libraries for Java support. Will try it out a bit later.

Great to learn something!
 
Last edited:

gkm

Expert Member
Joined
May 10, 2005
Messages
1,519
For something that involves IOT, you really should not use http polling. What you are doing there is fine for a handful of devices, but is not going to scale to thousands of devices.
 
Last edited:

SBSP

Senior Member
Joined
Sep 7, 2007
Messages
663
I think you are right.

I thought I had it working. I'm using session variables which I cant access from another session to change the switch status. So it will always report the last status in memory.

Don't you just love it when you build something and close to the end you realize its not going to work and its totally flawed!

Maybe I must just get a linux VPS and install mosquito MQTT. Hopefully I can find a way to consolidate my 2 existing domain names onto the VPS and still pay close what I'm paying now.
 

gkm

Expert Member
Joined
May 10, 2005
Messages
1,519
Yes, mosquitto is a good choice. It uses very little resource wise. It works well for these kinds of things.

Two tips I can think of:
- Remember to increase your file descriptors on the OS if you plan to connect a large number of devices.
- Never hardcode an IP address on the devices. Always use a DNS name. I have seen significant tears where an IP address was "lost" for some reason and then it bricked a bunch of devices that had to be recalled to be fixed.
 

SBSP

Senior Member
Joined
Sep 7, 2007
Messages
663
Ok so I registered with host1plus and got the silver package. Lets see how it goes :)
 

SBSP

Senior Member
Joined
Sep 7, 2007
Messages
663
Damn I let my self into more work than I initially anticipated!


Initially it was called griblinks.co.za but the hosting provider dropped me. Then I re-created the system and called it Griblinks - reloaded and placed it in a sub directory of a different domain
Here's my old system, Only database driven and only supports Raspberry Pi
http://www.techtinker.co.za/griblinks/


The all new and improved version with a shiny new domain :D
https://www.flyckerlink.com/

Its running a public mosquitto MQTT server. With Username: Demo and Password: Demo you can publish and subscribe to any topic you want.
I.E
Code:
mosquitto_sub -h flyckerlink.com -t "home/test" -u "Demo" -P "Demo"
I.E
Code:
mosquitto_pub -h flyckerlink.com -t "home/test" -m "TEST MESSAGE" -u "Demo" -P "Demo"

Next step is to get a new sketch developed for the micro controller and also the Flyckerlink switchpanel needs to publish an MQTT message when you click the toggle switch.

After that I need to make a new Android App, Have some cool features in mind that I want to add to the inline switch panel aswell, I.E Group Switching, Event notifications from the Microcontroller such , I.E Alarm triggered ect ect. I also want to add a config button that will generate the ESP8266 firmare for the people who are not familiar with programming stuff.


Thanks for the comments!
 

FarligOpptreden

Executive Member
Joined
Mar 5, 2007
Messages
5,396
I just wanted to recommend MQTT and Mosquitto, but I see you already went that route. It really makes communicating with IoT devices a breeze.
 

MDE

Expert Member
Joined
May 18, 2009
Messages
2,247
Small spelling error

"Schedules"
fa9024a321d2d02d595122f64f866346.jpg
 
Top