Programming/Logic quandry

zippy said:
This thread is an excellent example of how to over think and over engineer a problem which was solved probably back in the 70’s or early 80’s :)

Edit: Google “unix split command on windows”
Click to expand...
Depends on the workflow around the split; because to run an external command from C# and capture the return code is not a trivial matter in comparison with the code examples in this thread.

Aside from the fact that Windows doesn't have a split command; linux and macOS do.
 
If I could be naughty and do a hijack, I have also perhaps been overthinking this and not getting anywhere.

I have built a bagging machine and thought it would be nice to give the guy's a way of keeping tab's on it.

It runs on an ESP and when one of the scales tip it writes to a .csv file like this, date/time then the number of bags for today.

Code: 
1552334967,25,
1552334973,26,
1552338975,1,
1552338983,2,
1552338987,3,
1552338989,4,
1552338990,5,

I then use that and display it(on a webpage) nicely on a graph and it looks awesome :). Then our dear friends at Eskom **** around with the power and my counter gets reset.
So I thought, ok fine, let the graph look funny, I will just add a counter for today's and yesterday's production, because I just have to count the number of times a day appears in the array, choose yesterday and today, count and display.

But I have not succeeded (probably very silly attempts) with anything I have tried.

This was my starting point.

JavaScript: 
function parseCSV(string) {
    var array = []

    var lines = string.split("\n");
    for (var i = 0; i < lines.length; i++) {
        var data = lines[i].split(",", 2);
        data[0] = new Date(parseInt(data[0]) * 1000);
        data[1] = parseFloat(data[1]);
        array.push(data);
    }
    return array;

:) What it looks like now is too disgusting to publish!

Both ends( ie. what gets written to the file and what runs on the client) are under my control and can be changed, however if I change the .csv I will have to fix the graph portion :) (and that took me a long time to do.)

Is what I'm trying to do possible? should I put it in a JSON format and parse it out ? or perhaps something else?

I would prefer not to have to many writes to the ESP and obviously space is at a premium.
 
Have the esp rather POST to a system with more high level IO and power redundancy. Eg bag by bag or summary by the minute/hour
 
Spacerat said:
Have the esp rather POST to a system with more high level IO and power redundancy. Eg bag by bag or summary by the minute/hour
Click to expand...

Thanks for that, my thoughts were (and :) this is not my day job) that if I had the data stored on the ESP and then the client doing all the complicated stuff would be the way to go, because then any pc on the network could check in and see the numbers.

But maybe as you say a Pi of some sort..
Another option would be to put the esp on a power bank as a ups, but I am a fan of "Less moving parts" just means more bits that break.

So what I'm trying to do is not possible/simple in JS?
 
Düber said:
Thanks for that, my thoughts were (and :) this is not my day job) that if I had the data stored on the ESP and then the client doing all the complicated stuff would be the way to go, because then any pc on the network could check in and see the numbers.

But maybe as you say a Pi of some sort..
Another option would be to put the esp on a power bank as a ups, but I am a fan of "Less moving parts" just means more bits that break.

So what I'm trying to do is not possible/simple in JS?
Click to expand...

Digressing a bit from the original post. But IMHO the ESP is really only there to easily interface to a sensor. So in combination of the actual sensor, it becomes a smart sensor or IoT device. The purpose of the IoT device is to gather data and post it to a ‘server’ where all the data is stored, aggregated etc etc. The IoT device should not be doing all of that.
 
Spacerat said:
Digressing a bit from the original post. But IMHO the ESP is really only there to easily interface to a sensor. So in combination of the actual sensor, it becomes a smart sensor or IoT device. The purpose of the IoT device is to gather data and post it to a ‘server’ where all the data is stored, aggregated etc etc. The IoT device should not be doing all of that.
Click to expand...

I have found the ESP's to be capable little things, obviously not for complex stuff, but certainly able to send a few small files. I have just connected 5 things to it and they are all displaying the graph etc. fine. Anyhow as you pointed out I am digressing, but will certainly keep your suggestions in mind if the testing or actual working shows shortfalls.

However, I have got the "today and yesterday" counter's working in the javascript file and am very pleased, and do appreciate your replies.
 
gkm said:
Longer term, have a look at this kind of thing, for more robust long term reporting etc: https://www.google.com/search?q=aws+iot+esp8266
Click to expand...

Very interesting, as one of the things that is a problem is that after my files I have around 2.8MB left for the log to grow into(@ 5.7kB a day) so I am still deciding how to handle that. Tailing the file is easy, but it does not seem to be possible to remove the top lines of the file as far as I can tell.

So this might be a better option compared to having to go and either insert a blank file or manually giving it a haircut.

EDIT- not 2.8MB about 1.8
 
Last edited:
Still not sure about the over engineering in regards to memory is for if the I/O is the slowest part if this file is less than the memory limits.
Let's build a rocket that can go from point A to point B saving as much power as possible, meanwhile we have solar panels that generate enough power that if it doesn't go over the limit, no one cares. :p
 
Johnatan56 said:
Still not sure about the over engineering in regards to memory is for if the I/O is the slowest part if this file is less than the memory limits.
Let's build a rocket that can go from point A to point B saving as much power as possible, meanwhile we have solar panels that generate enough power that if it doesn't go over the limit, no one cares. :p
Click to expand...

Penny drop moment! (If there was a hit yourself on the head emoji)
Major rethink...um Thanks all.
 
One possible solution is to make a change to code that is generating the large file so that it doesn’t create one massive file, like some logging frameworks do. Ofc, that is only going to solve the problem going forward. Then do the unix/Linux/git bash split thing to solve the immediate needs. Seems like a bit of design flaw creating a massive file that is unusable.
 
Solarion said:
I think I need someone with a more Einstein way of thinking here.

Problem: I have a large text file with x amount of thousands of rows. Management wants this text file split into smaller text files of 500 rows. The last file might only have 50 or 80 or even 499 rows depending on the remainder of what is left in the main file.

This is the basic block of logic I have so far.

C#: 
public static void ReadFile(string file)
        {
            int y = 0;

            var lines = File.ReadAllLines(file);

            foreach (var line in lines)
            {
                y++;

                if(y == 500)
                {
                    y = 0; //each time this reaches 500 write to a new file and start the counter again.
                }
            }

        }

I'll bet you can see the problem already. What when I only have say 100 rows or even 499 rows left. How is my logic going to compensate for that remainder of rows in the file?

Perhaps I'm going about this all wrong in splitting up a text file. At this point I'm dead in the water.
Click to expand...

Why do they want to split this up?

I’ve seen decisions to split things up destroy performance quite a few times. File systems prefer fewer files, read efficiency prefers contiguous data.
 
cguy said:
Why do they want to split this up?

I’ve seen decisions to split things up destroy performance quite a few times. File systems prefer fewer files, read efficiency prefers contiguous data.
Click to expand...
Dunno ... maybe for upload purposes
 
cguy said:
Why do they want to split this up?

I’ve seen decisions to split things up destroy performance quite a few times. File systems prefer fewer files, read efficiency prefers contiguous data.
Click to expand...
This would be balance between parsing and file I/O time, use case dependent. If you can split by e.g. day, then that will probably make it more efficient, again, use case dependent.
 
cguy said:
Why do they want to split this up?

I’ve seen decisions to split things up destroy performance quite a few times. File systems prefer fewer files, read efficiency prefers contiguous data.
Click to expand...

The third party software wasnt designed to handle such massive input at any one time. The customer base has grown so massive that a workaround had to be found. Its working now and the system no longer times out.
 
eye_suc said:
The amount of lines were known up front, right?
Could also have tried a normal for loop (i.e. for (int i = 0; i < lines.Count(); i++)) and then using Modulus (i % 500) to determine when to switch to a new file.

Glad you got sorted :)
Click to expand...

This is the first thing I thought of.
 
zippy said:
This thread is an excellent example of how to over think and over engineer a problem which was solved probably back in the 70’s or early 80’s :)

Edit: Google “unix split command on windows”
Click to expand...

Not everyone knows how to use BASH and GREP.
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X