IoT Drinking Birds

While at Stack41, I created a demonstration of our IoT service using drinking birds. The following is a draft article I wrote while employed there in 2019; it describes the technology a bit. I built two other demonstrations of our platform, but are not documented here: a beer tap counter, and a CO2 fizzy-watercooler rig.

Introduction

Caravela, on top of Stack41 services, is able to run a private IoT cloud atop Free Software (not in cost, but in permissive-licensing). In order to bring data into our data center, we needed a cellular gateway device. We didn’t like anything that already existed in the market, so we created our own.

We quickly found ourselves engaged with a water treatment customer, but the engagement started taking far too long, and the project eventually stalled. We have tried to get further engagements, but find that many people can’t comprehend what the device is, or how it works. To help explain this, our crack engineer Davis devised a little science experiment that provides ample data to showcase our services. A desk toy can provide useful data, so drinking birds were used. There was no further rationale behind this decision of object other than “because it’s fun.” By reading the metrics of a drinking bird feeder system, we are able to display metrics in an online dashboard that can be further extended with deeper queries and email alerting.

Constraints and Design

In order to devise this machine, we needed to consider many constraints. First, the issue of privacy is most dear to Davis, so there was no way he’d let go of a customer’s data for a public demonstration. In Davis took the initiative to build the model on his own. He had rattled the idea around before, months before actually, but it seemed like the right time to build a project. Drinking birds are not just desk toys; they are heat engines. Most of the staff did not see the immediate usefulness of playing with them, but Davis was determined to prove them otherwise.

Budget is always a factor, and since Davis was doing this “in the dark,” he couldn’t spend a lot of money. With three trips to American Science and Surplus, and fewer than $90 pocketed, he wandered the store looking for artifacts of inspiration. He knew that we needed to measure the periodic motion of the birds, and knew that tracking water could be useful, so he looked through the containers first.

Davis found two big containers (probably 1.5 L) that would serve nicely to feed the rest of the system. These are the tanks that need to be refilled by a human. After settling on these, he knew the birds would each need their own feeder tank to keep them swimming. So, four more containers are needed that fit that requirement. For a couple dimes a piece, he found some jars.

With primary tanks, and smaller feeder tanks, Davis needed a way to move water from one to the other. Stopping in the electro-mechanical isle, he found some little pumps that run on 5 VDC, and only pull about 200 mA when working. They were somewhat pricey, at $7.50 per pump, so then the decision needed to be made: should the model use four pumps, or one pump and a selector valve? Well, the latter seemed cheaper, so he walked away with a single pump after the first visit. However, quickly he realized it wasn’t worth the hassle of designing and sourcing parts for a selector valve, so that idea was shelved and Davis bought a few more of them.

Now, Davis had two primary tanks, four feeder tanks, four birds, and four pumps. He snagged some infrared photo-transistors from Sparkfun, which serve as optical “bird drinking” detectors. Optical was best used here, because the birds are delicate and balanced, and any contact would disrupt its periodic motion. This worked, at first, but since glass only allows a specular reflection, the gain needed was increased to detect the motion. But, doing this also highlights background noise, which can happen if one gets too close to the sensor with their hand, or if it is a particularly sunny day. This limitation was solved by sticking a piece of envelope to each bird’s bulb, increasing diffuse reflectivity.

In order to move water, he connected the parts with 1/4” PVC tubing, which is both flexible and optically clear (a nice touch for humans looking at it). The two primary tanks were modified with 3/8” vinyl outlet ports, which mate well with the quarter-inch tube. The tube also mates well with the pumps after a little treatment with a hot-air gun. In order to get the water into the feeder tanks, more 3/8” vinyl tube was used, but one end was sealed and a hole drilled to redirect the flow 90 degrees. This worked pretty well, and the tube is fastened to the feeder tanks with binder clips.

The two primary tanks, in this configuration, need to get their water from somewhere. Davis thought an excellent demonstration of our product would capture real-time alerts. So, he made a low-water sensor, which is triggered to command a human to re-fill the tanks via email. This was created with some plastic string (resisting mold and mildew), a ping pong ball, and a micro-switch. When the water level is low, gravity pulls the ball low, activating the switch. When the water is filled back up, buoyancy out-pulls gravity, and the switch is deactivated. A small nut provides ballast weight.

The model is wired into an Arudino: a micro-controller platform well-embraced by hobbyists. Because the TBox, our data-acquisition appliance, speaks Ethernet, so must the Arduino. Using this common interface lets us talk to anything else that speaks IP. WizNet modules are cool, because they handle the entire IP stack and speak SPI, making it a breeze to use. A code library was already available, and we had it talking on our network in fewer than twenty minutes. We programmed code into the Arduino, making it a mini-HTTP server, so it also serves as a useful example of the kinds of things TBox can talk to. The server, upon any request, will return a list of tags: names of properties in the machine, and their values. Tags look like this:

and are composed of names and values. Not only could tags read the values of sensors, they may also contain intermediate calculations held in the machine’s mind, which bear no familiarity. An example of the weird kind are internal timers—a number that keeps increasing, a kind of ticking clock—to orchestrate larger events. The idea of tag acquisition is behind Caravela’s IoT cloud, because it is the Rosetta format for transmitting factory statistics. Our TBox is connected to both the drinking bird controller, and an Allen-Bradley ControlLogix 5500 PLC, and is reading data from both continuously.

The PLC is in our conference room, and doesn’t actually do anything, so those tags are only for the purpose of demonstration. We used to have it hooked up to beer taps (with tilt switches) but that demonstration didn’t work very well because the tilt switches wouldn’t keep attached to the beer taps. Wire gauge was partially to blame, for stiffening the connection to the switch. The tilt switches, themselves, were also to blame; their quality was what one could expect out of an eBay-Shenzhen sale.

Because our TBox appliance uses completely open software, we are free to modify it as we like. This is Free, as in Freedom. We are not bound by restrictive software licenses, and our software is completely non-proprietary. We encourage anyone who gets a TBox to ask us for the source code, because it’s their right. And, once they have the source code, they are Free to do anything they want with it. No single entity owns it, and this allows us to stand on the shoulders of giants, and provide really neat features.

Free Software

Software sometimes costs money, and Free Software is no exception. The word Free does not refer to price, but rather Freedom: what one can do with a program, once they have it. By using Free Software, we are obligated to release our modifications under the same license, making it viral. We are also not afraid to show people our code, because we take pride in shining light on the ugly parts. This lets us recognize the bad parts for what they are, fix them, and move on. Why won’t proprietary companies tell their customers how their machines and programs work? Who’s to say they do work, when nobody outside the organization can attest to it? The same line of thought goes with security: how can proprietary companies say their products are secure, without ever letting the public look at them?

Free Software is not just important because eyes catch problems; it is also important because it gives the user the freedom to do whatever they want with a program, including the right to modify and redistribute new copies, without any license restrictions or royalty fees. One who is in possession of Free Software can do anything they want with it, so long as they release their derivative as Free Software also. This has been tremendously useful at Caravela and Stack41, and our business wouldn’t be possible without the fruit of Free Software.. We owe it to our customers to offer our software under the same licenses we use, and they can reap the rewards with us.

About TBox

The TBox appliance is a hardware device that sits on a network, and asks all the other computers, “how are you doing?” When the devices respond, with tags, the TBox transmits those back to the data center where they are stored in a customer’s database. This isn’t just any database, though: it’s PostgreSQL, with TimescaleDB as an extension, which makes querying billions of tag samples fast. This is very helpful when dealing with signals over time, such as those which come from PLCs and other industrial equipment. But, it also works great for reading the values of drinking birds.

The TBox runs a modified Alpine Linux operating system. This was chosen because it can boot-to-RAM by default, which increases the TBox’s SSD life by decreasing read and write operations. The only time the SSD is used is under software upgrades, or when the connection to the remote database is lost and the data needs to be buffered locally. Being designed around a cellular connection, the TBox is very fault-tolerant.

On the other side, in the data center, each customer gets their own private VLAN where their services reside. This makes it more-secure, because if one customer’s network is breached, neighboring tenants are completely isolated. Each customer also gets their own public IP addresses, so nobody shares their IoT cloud with another.

Inside the cloud, each tenant is automatically provisioned two firewalls (pfSense in CARP configuration) that run on separate physical hosts. These firewalls let good people in to the IoT network, and keep bad people out. VPN connections, through OpenVPN, are established between the TBox and these firewalls, and they are not allowed to talk to anything else. Services behind these firewalls, such as databases (with redundancy) or web-based analytic software, may only speak to other approved services. We believe this is a responsible design of IoT infrastructure, and is only possible because Caravela owns their hardware.

Analytics

One piece of Free Software we like is PostgreSQL, because it is a proven database system that is both performant and free-in-cost. It’s lovely to have a SQL interface to all one’s data: questions are easy to form, and answers are often keystrokes away. Not only that, but other programs often speak SQL, so integration is low-effort.

One program we use, that integrates flawlessly, is Grafana: a charting, analytics, alerting and reporting package that runs as an application server. It works so well with the database, that making a dashboard to showcase the drinking birds only took a day, and another few days to make it pretty, with alerting. That time was mostly spent writing queries, not connecting the two.

Grafana is always running, and can alert users when something happens with a machine. We use it to email us, reminding us to refill the primary water tanks when they get low. That is the only maintenance work that needs to be done to our model, but it can tell us when something goes wrong, too. When a bird is not oscillating, but the tanks are full, then we can throw an alert to an “engineer” who can further look at the machine, and diagnose the problem. This keeps the model hands-off, just like a Plant Manager might want of their machines.

Of course, since all our stuff works with open protocols, a customer could even use proprietary software (such as Tableau) with our cloud. It doesn’t make a difference to the database.

This is a diagram about how the IoT cloud works, from a high-level perspective. Different connection types are defined with lines, and the objects are labeled in-sequence.

The Journey of a Tag

What is a ‘tag’, and how does it have a journey?

Usually, humans want to measure their machines, and have an action occur if something happens. This is very vague, but so are the problems that computers solve. Whatever that something is, a sensor converts the physical happening into an electrical signal. Such a happening could be when a drinking bird goes down to get a taste of water, an electrical signal is based on the number of photons bouncing off the bulb.

This electrical signal is read by a kind of computer, called a programmable logic controller (PLC). The PLC keeps track of how many times signals have happened, and can be programmed (with logic) to control outputs like pumps and motors. The PLC has many inputs, and can make many other numbers calculated from them. These numbers, along with their names, are called tags. The PLC usually needs other supporting devices, so it can’t keep its tags to itself. So, other devices on that LAN can ask for them.

The TBox is always connected to the Internet with a cellular data link. It knows not to talk to anybody on the Internet, except for Stack41. The first thing the box does when booting is start a VPN tunnel, so it can talk (only) to the database in the data center. If this is not available, then the TBox will buffer tags locally, until a link is re-established. The TBox is on the same LAN as the PLC, and therefore can talk to it.

The TBox asks the PLC for its tags at regular intervals. Usually, this is every five seconds, but this sample rate is entirely configurable. The PLC responds back with the tags, and the TBox will try to put them into the database. If the TBox can’t fulfill that, for whatever reason (cellular signal is poor, database is full, etc), then it will store them locally until the secure link is re-established, then upload them while continuing operations as normal. To a human looking at the database through an application server, like Grafana or Tableau, live data would be temporarily unavailable during an outage, but would fill-in once communication is re-established.

Once the tags are in the database, a number of application servers can do all kinds of nifty calculations and trigger actions. One server we like the most is Grafana because it lets someone create pretty dashboards from their tag data, and trigger alerts based on any given logic. So, if the database says that a water tank is running low, an email can be dispatched to give the operator instructions to re-fill it. The operator can also look to this data when making decisions about how to proceed, if multiple parts of the system are detected as malfunctioning. Operators can also query the database to know when things are going to happen, such as, “based on the rate of water flowing through Pump A and Pump B, how long will it take before someone needs to fill Tank A?” This kind of information is useful for scheduling maintenance in-advance.

The journey of a tag doesn’t need to end here; application servers can do literally anything. It is only limited by the imagination and willingness of the programmer to tell it what to do. Many Free Software packages exist that Caravela can host with no license fees. But, if someone has software they really want to use, we can host it in a secure, virtualized environment. And, our customers now their data is never locked into proprietary software; everyone is free to export their data any time they like.