6

I am building a monitoring system for an irrigation pump and the connected pipeline. So far I have completed the sensor network. Everything connects to a micro-controller board (actually 4 Teensy 3.0 boards) and the board is programmed to output a string of formatted data (the readings) at predetermined intervals.

What I want to do is connect the board via USB-serial to a RaspberryPI (or some other SBC) and have the computer send the readings it receives from the sensor board to a remote server for logging.

The options I have considered are:

1 - Syslog... It struck me early on as a possibility because it seems to offer pretty much everything I need. The complexity scares me though.

2 - REST... Have the RaspberryPI POST the data, line by line, over the net to CouchDB on the server.

3 - HTTP... Keep an open HTTP connection with node.js and "write" the lines of data. It will of course need to be received by a second node.js script for storing in the DB.

Now to the requirements:

~ It needs to be lightweight and relatively fast. There will be a lot of data (1s intervals) and the RaspberryPI is no powerhouse.

~ I would very much like the option for a compressed string. The uplink is via 3G and I hope to go with a "small" monthly plan.

~ Encryption will be nice, but not mandatory. Paranoia is strong with the rednecks...

~ I really need this to be as K.I.S.S. as possible.

To cut a long story short, I have though of this as some kind of over-the-net serial connection, where a computer feeds line after line to another computer.

So, whitch of the options I have here is preferable? Or even better, has anyone a better idea?

I am honestly open to editing and even reposting the question if someone has a good point to make.

EDIT:

All comments and answers so far have been appreciated and considered.

Syslog is really great, but I really need to avoid the complexity and overhead. Also, after some testing RaspberryPI seems to grind to a halt shortly after starting rsyslog.

By now it has been decided that the DBMS is going to be CouchDB.

The obvious option would be to use curl or a vestigial node.js server to make the REST calls to the DB server immediately as soon as the data arrives. This, although simple and effective, is for a number of reasons not preferable. Security is also a concern; I do not like the idea of a miniature PC in a field making direct calls to the DBMS.

The reason I am starting a bounty is that I hope someone can suggest an idea along the following lines: "Some kind of persistent connection between the remote miniature PC and the DBMS. Data will be formatted by some kind of protocol and forwarded through this connection in order to be received at the opposite end. This connection needs to be as lightweight as possible, with the least possible amount of overhead".

7
  • 1
    1 is far more KISS than 2 and 3.
    – fukawi2
    Sep 13, 2013 at 1:00
  • I do a similar thing; accepting text from serial input, sending to a remote machine (internationally), I use AMQP/RabbitMQ to handle it because I need to action the data on the remote end, but if you're just after logging it, using a syslog variant is almost certainly the least complex.
    – fukawi2
    Sep 13, 2013 at 1:04
  • 3
    Personally I would go for REST. It's a well known and well understood protocol. You can compress it (post gzipped data), you can encrypt it (SSL). It's simple to write both sides of the connection. You can send basically anything you want - as a CSV, as an XML, etc. Depending on how you code it you can send it one line at a time, or in batches. Sep 13, 2013 at 1:17
  • 2
    syslog is too stupid and harder to integrate, syslog-to-syslog even with RELP is not in KISS. REST - why to complicate HTTP? HTTP - most supported, easier to manipulate, but requires stable connection, queuing. So, question is what do you do on the other side? If you don't care, then see the answer.
    – GioMac
    Sep 13, 2013 at 2:41
  • 1
    No idea about couchdb - but also not sure how that fits into your model - I'm not suggesting posting straight from remote into the couchdb - you have an interim step in the form of a web server and a web client on opposite ends that will do that for you Sep 13, 2013 at 3:01

3 Answers 3

4
+50

Have you thought of using SMTP (email) ? The Raspberry has two processes on it: one reads data and append them to a file, another one (possibly in crontab) moves the files (possibly aggregating some) and send them by email to the destination machine. This way, you have no coupling between the frequency of sampling and the frquency of logging, if it's okay with you (by sending an email every 20s or every minute for example).

The mail can be compressed, encrypted (using SMTP TLS). Moreover it's resilient: in case you lost your uplink, the data we'll be sent when the link is restored. From your logging process point of view, the link is always "up".

On your couchdb server (or another machine connected to the couchdb), you create a dedicated user and put a script in its .forward that unzip the message and feed it to couchdb.

If you want to authenticate, you can go with many schemes from a shared secret to PGP signing !

We do this to feed our data warehouse, because we don't want any kind of incoming connection on it (https or ssh), admittedly not at a 1s sampling rate (but with bigger data).

Last but not least, you can test every component individually (logger, sender, receiver and db-feeder) without needing all the infrastructure running.

2
  • This what I was looking for! Simple, fast, versatile... SMTP being asynchronous is a nice bonus. I do not really care if the data is logged every 1s, only that I actually HAVE new data every 1s. You shall have my +50 when the bounty expires. Now I have to go read SMTP.
    – dlyk1988
    Sep 17, 2013 at 23:38
  • I'm glad if I was able to help. Don't hesitate to add comments if you have questions about configuration, I can also provide snippets of code (python or perl). Sep 18, 2013 at 8:26
2

Generally, everything depends on what you do on the other side. For example: if you need this data to be forwarded to the Zabbix - you use Zabbix agent, for SNMP - you use snmpd, for web application processing - HTTP etc.

Rsyslog could be a good option for transport, because it can already solve many problems you may encounter while developing custom solution (i.e. when working with REST, generic HTTP):

  1. It's reliable with RELP
  2. Has on-disk queuing option, so you can deliver messages after connection is re-established
  3. Supports gzip message compression
  4. Supports TLS
  5. Supports multiple input modules
  6. And output modules
  7. Filtering and modification on both sides

TLS will have huge overhead - try to not use it at all.

You will need to setup same system on the other side, then you can do tricks with messages as you want i.e. insert it to the mysql or/and postgresql database, or/and any other with DBI, or/and forward to file, or/and to named pipe, or/and as snmp trap, or/and as email, and/or to custom application etc.

I'm using similar system that you have (Arduino+Raspberry), but I'm running proprietary application with it's specific queue and SOAP transport. For simple messaging, I think I'd use rsyslog.

1

You could also look into RabbitMQ Since it's a message broker, which is kind of designed for this problem.

RabbitMQ is a message broker. The principal idea is pretty simple: it accepts and forwards messages. You can think about it as a post office: when you send mail to the post box you're pretty sure that Mr. Postman will eventually deliver the mail to your recipient. Using this metaphor RabbitMQ is a post box, a post office and a postman.

The major difference between RabbitMQ and the post office is the fact that it doesn't deal with paper, instead it accepts, stores and forwards binary blobs of data ‒ messages.

Here's a post describing how to use it on an Rpi.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .