One of the projects I’m working on involves creating a little device which you talk to from your phone. So, I thought, I’ll do this properly. No “cloud service” that you don’t need; no native app that you don’t need; you’ll just send data from your phone to it, locally, and if the owners go bust it won’t brick all your devices. I think a lot of people want their devices to live on beyond the company that sold them, and they want their devices to be under their own control, and they want to be able to do all this from any device of their choosing; their phone, their laptop, whatever. An awful lot of devices don’t do some or all of that, and perhaps we can do better. That is, here’s the summary of that as a sort of guiding principle, which we’re going to try to do:
You should be able to communicate a few hundred KB of data to the device locally, without needing a cloud service by using a web app rather than a native app from an Android phone.
Here’s why that doesn’t work. Android and Chrome, I am very disappointed in you.
The first reaction here is to use Bluetooth LE. This is what it’s for; it’s easy to use, phones support it, Chrome on Android has Web Bluetooth, everything’s gravy, right?
No, sadly. Because of the “a few hundred KB of data” requirement. This is, honestly, not a lot of data; a few hundred kilobytes at most. However… that’s too much for poor old Bluetooth LE. An excellent article from AIM Consulting goes into this in a little detail and there’s a much more detailed article from Novelbits, but transferring tens or hundreds of KB of data over BLE just isn’t practical. Maybe you can get speeds of a few hundred kilo bits per second in theory, but in practice it’s nothing like that; I was getting speeds of twenty bytes per second, which is utterly unhelpful. Sure, maybe it can be more efficient than that, but it’s just never going to be fast enough: nobody’s going to want to send a 40KB image and wait three minutes for it to do so. BLE’s good for small amounts of data; not for even medium amounts.
WiFi to your local AP
The next idea, therefore, is to connect the device to the wifi router in your house. This is how most IoT devices work; you teach them about your wifi network and they connect to it. But… how do you teach them that? Normally, you put them in some sort of “setup” mode and the device creates its own wifi network, and then you connect your phone to that, teach it about your wifi network, and then it stops its own AP and connects to yours instead. This is maybe OK if the device never moves from your house and it only has one wifi network to connect to; it’s terrible if it’s something that moves around to different places. But you still need to connect to its private AP first to do that setup, and so let’s talk about that.
WiFi to the device
The device creates its own WiFi network; it becomes a wifi router. You then connect your phone to it, and then you can talk to it. The device can even be a web server, so you can load the controlling web app from the device itself. This is ideal; exactly what I planned.
Except it doesn’t work, and as far as I can tell it’s Android’s fault. Bah humbug.
You see, the device’s wifi network obviously doesn’t have a route to the internet. So, when you connect your phone to it, Android says “hey! there’s no route to the internet here! this wifi network sucks and clearly you don’t want to be connected to it!” and, after ten seconds or so, disconnects you. Boom. You have no chance to use the web app on the device to configure the device, because Android (10, at least) disconnects you from the device’s wifi network before you can do so.
Now, there is the concept of a “captive portal”. This is the thing you get in hotels and airports and so on, where you have to fill in some details or pay some money or something to be able to use the wifi; what happens is that all web accesses get redirected to the captive portal page where you do or pay whatever’s necessary and then the network suddenly becomes able to access the internet. Android will helpfully detect these networks and show you that captive portal login page so you can sign in. Can we have our device be a captive portal?
No. Well, we can, but it doesn’t help.
You see, Android shows you the captive portal login page in a special cut-down “browser”. This captive portal browser (Apple calls it a CNA, for Captive Network Assistant, so I shall too… but we’re not talking about iOS here, which is an entirely different kettle of fish for a different article), this CNA isn’t really a browser. Obviously, our IoT device can’t provide a route to the internet; it’s not that it has one but won’t let you see it, like a hotel; it doesn’t have one at all. So you can’t fill anything into the CNA that will make that happen. If you try to switch back to the real browser in order to access the website being served from the device, Android says “aha, you closed the CNA and there’s still no route to the internet!” and disconnects you from the device wifi. That doesn’t work.
You can’t open a page in the real browser from the CNA, either. You used to be able to do some shenanigans with a link pointing to an
intent:// URL but that doesn’t work any more.
Maybe we can run the whole web app inside the CNA? I mean, it’s a web browser, right? Not an ideal user experience, but it might be OK.
<input type="file"> elements don’t work; you can’t open a file picker to allow someone to choose a file to upload to the device. So that’s a non-starter too.
So, what do we do?
Unfortunately, it seems that the plan:
communicate a few hundred KB of data to the device locally, without needing a cloud service by using a web app rather than a native app from an Android phone
isn’t possible. It could be, but it isn’t; there are roadblocks in the way. So building the sort of IoT device which ought to exist isn’t actually possible, thanks very much Android. Thandroid. We have to compromise on one of the key points.
If you’re only communicating small amounts of data, then you can use Bluetooth LE for this. Sadly, this is not something you can really choose to compromise on; if your device plan only needs small volumes, great, but if it needs more then likely you can’t say “we just won’t send that data”. So that’s a no-go.
You can use a cloud service. That is: you teach the device about the local wifi network and then it talks to your cloud servers, and so does your phone; all data is round-tripped through those cloud servers. This is stupid: if the cloud servers go away, the device is a brick. Yes, lots of companies do this, but part of the reason they do it is that they want to be able to control whether you can access a device you’ve bought by running the connection via their own servers, so they can charge you subscription money for it. If you’re not doing that, then the servers are a constant ongoing cost and you can’t ever shut them down. And it’s a poor model, and aggressively consumer-hostile, to require someone to continue paying you to use a thing they purchased. Not doing that. Communication should be local; the device is in my house, I’m in my house, why the hell should talking to it require going via a server on the other side of the world?
You can use a native app. Native apps can avoid the whole “this wifi network has no internet access so I will disconnect you from it for your own good” approach by calling various native APIs in the connectivity manager. A web app can’t do this. So you’re somewhat forced into using a native app even though you really shouldn’t have to.
Or you can use something other than Android; iOS, it seems, has a workaround although it’s a bit dodgy.
None of these are good answers. Currently I’m looking at building native apps, which I really don’t think I should have to do; this is exactly the sort of thing that the web should be good at, and is available on every platform and to everyone, and I can’t use the web for it because a bunch of decisions have been taken to prevent that. There are good reasons for those decisions, certainly; I want my phone to be helpful when I’m on some stupid hotel wifi with a signin. But it’s also breaking a perfectly legitimate use case and forcing me to use native apps rather than the web.
Unless I’m wrong? If I am… this is where you tell me how to do it. Something with a pleasant user experience, that non-technical people can easily do. If it doesn’t match that, I ain’t doin’ it, just to warn you. But if you know how this can be done to meet my list of criteria, I’m happy to listen.