Best technologies to use with receiving data - insert/update - sending data?

M

MisterBigglesworth

Expert Member
Joined
Aug 15, 2006
Messages
3,334
Reaction score
13
Location
/\\/¯¯¯\//\
Howzit!

So this is new grounds for me. I have an idea for a system and I have a basic idea of what needs to be done. Simply put my system would run on/as a website and would need to read data from other website/s, take the data and manipulate as needed and then insert/update into my database and then finally send the data to the originating website and/or other websites.

Now I'm primarily a PHP / Classic ASP with SQL Server / MySQL developer (I have some basic experience in Java, and quite a bit with JS). I have a feeling these techs, excl JS, are not going to cut it. A lot of the Java devs I worked with recently raved about JSON....so I'm looking at this for the sending / receiving of data. The requirement is that the sending and receiving of data needs to be as quick and slim as possible, hence the choice of JSON.

For the database...I really have no idea. I know I dont want to use SQL Server....and I find MySQL at times to be difficult to work with when maintaining a database with many records.

My system will be running on a website, but I would like to expand to the Mobile and/or App route in the near future. So here I am...not sure what language to use, or database?

I'm leaning towards:

  1. Data Sending/Receiving between my system and other websites/systems - JSON
  2. Database - No idea
  3. Language - Python / Perl

I could really use some input here. Not afraid to take on learning a new language and database, I want to make the right decision from the start and set up a good framework. So if there is a huge learning curve involved, its all good with me. Struggle now, benefit later :p
 
Just try JSON, if it doesn't match your expectations, substitute it for something else.
Ditto for DB.

It should only take minutes to substitute
 
Thanks for the suggestion.

Didnt want to go SQL Server as I would prefer the open source route and a solution that works well together, if that makes sense? Come to think of it...I cant really give a very good answer to your question. Hmmm....
 
LOL was just curious actually. Most applications should be able to work with SQL Server and can't think of a reason to want to go with an open source database. Personally, I would prefer the best database. Not saying MSSQL is the best, just saying it's a good choice unless there was a valid reason not to use it e.g. hosting on Linux.
 
Well licensing cost for one. 90% of all database requirements could be handled by a low cost MySQL server. Host on rack space or amazon, and you get free redundancy, and more performance than you would ever need for the fraction of a mssql license.
 
Open source isn't always the best solution. Be aware of the licence conditions. A lot of companies avoid open source, especially software dev companies.

For example, Apple used to ship the GPL gcc compiler with its Xcode IDE. They stopped doing this because it forced all apple store developers to use the GPL licence, because the apps where compiled with a compiler on the GPL licence.

Just be aware of it :)

On choosing a db, use the best one for the OS that you can afford. Choose the db that you have the skills for.
 
PostgresSQL or Firebird for the DB? if you don't fancy MySQL and want to stay open source.

One of the proprietory systems that I work with has an XML messaging system which can be used to communicate with 3rd party applications for B2B etc. JSON could work too. I'm sure php can work fine with JSON.
 
JSON is a format, not a protocol. you can use it just as well in php, as writing it on paper and mailing to via the post office :)
 
Last edited:
JSON has a much much lighter payload than XML, so I think it's a great choice (at one client the use of JSON serialization instead of XML serialization in WCF service had such a great benefit it saved a project that was almost canned for network performance reasons).

All the major languages should support it with little to no effort.

You might also want to consider if you need security, and how to do that with your technology choice (e.g. OAuth).

I know you mentioned open source, but I'm going to suggest Microsoft as a possible alternative too, yes :D (Hey, they have been moving towards a more open source philosophy...). I'm suggesting, Windows Azure.

ASP.NET WebApi supports JSON, and also has the ability to support different content-types as well depending on the accept header sent through via the client / user agent. Deploying a web app/site is very very simple, and you don't have to worry about the infrastructure. Think they even have git support and deployments for simple web sites (not sure if this is only plain old web sites serving static content, or fully blown web applications with a service backend, it appeared to be the simple case only last I looked).

The cost for DB's on SQL Azure is quite low, however it doesn't have all the features of SQL Server (e.g. no access to proper file system means bulk import is a little bit of a pain).

AFAIK, they also have tooling for various environments like Linux and MacOS. But probably will get the best experience on a Windows environment, plus you can get the IDE etc. free if you use the Express editions.

I'm not sure about monthly fees and that, but in terms of simplicity, MS has done quite a nice job with Azure. It can be frustrating however as things can sometimes not be supported (e.g. bulk insert in SQL Azure), or can take a while to happen since it's in the cloud (e.g. if you setup a web role, a whole new VM is installed and setup for you on deployment!).

Perhaps there are other cloud based options as well that you could investigate, e.g. Heroku, and like _kabal_ kinda pointed to Amazon. Could be the way to go for a startup app, especially if you hit onto a winning idea and need infrastructure/scale.
 
If you're going to be using JSON (recommended), then why not have MongoDB as your DB?
 
Don't over-complicate these things:

LAMP: PHP/MySQL

cURL to scrape the websites with some regex (unless the websites you want to harvest data from actually has an available API to call, be prepared to work with XML/JSON/Arrays as their data sources, you can't control what they give you, but you can control what you give others)

cron to schedule the cURL (otherwise let Google-bot do the dirty crawl work for you if you have a set list of keywords they can crawl with and do the API calls as and when needed)
 
Python, PostgreSQL, JSON.

Use SQLAlchemy as your ORM layer and then depending on how you are planning on accessing other sites' data the Requests library will make life much easier.
 
the tech you use should always be complimented by your skillset, otherwise you end up making mistakes that could cost you in terms of performance, time & money
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X