IoT devices and Android and disappointment

One of the projects I’m working on involves creating a little device which you talk to from your phone. So, I thought, I’ll do this properly. No “cloud service” that you don’t need; no native app that you don’t need; you’ll just send data from your phone to it, locally, and if the owners go bust it won’t brick all your devices. I think a lot of people want their devices to live on beyond the company that sold them, and they want their devices to be under their own control, and they want to be able to do all this from any device of their choosing; their phone, their laptop, whatever. An awful lot of devices don’t do some or all of that, and perhaps we can do better. That is, here’s the summary of that as a sort of guiding principle, which we’re going to try to do:

You should be able to communicate a few hundred KB of data to the device locally, without needing a cloud service by using a web app rather than a native app from an Android phone.

Here’s why that doesn’t work. Android and Chrome, I am very disappointed in you.

Bluetooth LE

The first reaction here is to use Bluetooth LE. This is what it’s for; it’s easy to use, phones support it, Chrome on Android has Web Bluetooth, everything’s gravy, right?

No, sadly. Because of the “a few hundred KB of data” requirement. This is, honestly, not a lot of data; a few hundred kilobytes at most. However… that’s too much for poor old Bluetooth LE. An excellent article from AIM Consulting goes into this in a little detail and there’s a much more detailed article from Novelbits, but transferring tens or hundreds of KB of data over BLE just isn’t practical. Maybe you can get speeds of a few hundred kilo bits per second in theory, but in practice it’s nothing like that; I was getting speeds of twenty bytes per second, which is utterly unhelpful. Sure, maybe it can be more efficient than that, but it’s just never going to be fast enough: nobody’s going to want to send a 40KB image and wait three minutes for it to do so. BLE’s good for small amounts of data; not for even medium amounts.

WiFi to your local AP

The next idea, therefore, is to connect the device to the wifi router in your house. This is how most IoT devices work; you teach them about your wifi network and they connect to it. But… how do you teach them that? Normally, you put them in some sort of “setup” mode and the device creates its own wifi network, and then you connect your phone to that, teach it about your wifi network, and then it stops its own AP and connects to yours instead. This is maybe OK if the device never moves from your house and it only has one wifi network to connect to; it’s terrible if it’s something that moves around to different places. But you still need to connect to its private AP first to do that setup, and so let’s talk about that.

WiFi to the device

The device creates its own WiFi network; it becomes a wifi router. You then connect your phone to it, and then you can talk to it. The device can even be a web server, so you can load the controlling web app from the device itself. This is ideal; exactly what I planned.

Except it doesn’t work, and as far as I can tell it’s Android’s fault. Bah humbug.

You see, the device’s wifi network obviously doesn’t have a route to the internet. So, when you connect your phone to it, Android says “hey! there’s no route to the internet here! this wifi network sucks and clearly you don’t want to be connected to it!” and, after ten seconds or so, disconnects you. Boom. You have no chance to use the web app on the device to configure the device, because Android (10, at least) disconnects you from the device’s wifi network before you can do so.

Now, there is the concept of a “captive portal”. This is the thing you get in hotels and airports and so on, where you have to fill in some details or pay some money or something to be able to use the wifi; what happens is that all web accesses get redirected to the captive portal page where you do or pay whatever’s necessary and then the network suddenly becomes able to access the internet. Android will helpfully detect these networks and show you that captive portal login page so you can sign in. Can we have our device be a captive portal?

No. Well, we can, but it doesn’t help.

You see, Android shows you the captive portal login page in a special cut-down “browser”. This captive portal browser (Apple calls it a CNA, for Captive Network Assistant, so I shall too… but we’re not talking about iOS here, which is an entirely different kettle of fish for a different article), this CNA isn’t really a browser. Obviously, our IoT device can’t provide a route to the internet; it’s not that it has one but won’t let you see it, like a hotel; it doesn’t have one at all. So you can’t fill anything into the CNA that will make that happen. If you try to switch back to the real browser in order to access the website being served from the device, Android says “aha, you closed the CNA and there’s still no route to the internet!” and disconnects you from the device wifi. That doesn’t work.

You can’t open a page in the real browser from the CNA, either. You used to be able to do some shenanigans with a link pointing to an intent:// URL but that doesn’t work any more.

Maybe we can run the whole web app inside the CNA? I mean, it’s a web browser, right? Not an ideal user experience, but it might be OK.

Nope. The CNA is a browser, but half of the features are turned off. There are a bunch of JavaScript APIs you don’t have access to, but the key thing for our purposes is that <input type="file"> elements don’t work; you can’t open a file picker to allow someone to choose a file to upload to the device. So that’s a non-starter too.

So, what do we do?

Unfortunately, it seems that the plan:

communicate a few hundred KB of data to the device locally, without needing a cloud service by using a web app rather than a native app from an Android phone

isn’t possible. It could be, but it isn’t; there are roadblocks in the way. So building the sort of IoT device which ought to exist isn’t actually possible, thanks very much Android. Thandroid. We have to compromise on one of the key points.

If you’re only communicating small amounts of data, then you can use Bluetooth LE for this. Sadly, this is not something you can really choose to compromise on; if your device plan only needs small volumes, great, but if it needs more then likely you can’t say “we just won’t send that data”. So that’s a no-go.

You can use a cloud service. That is: you teach the device about the local wifi network and then it talks to your cloud servers, and so does your phone; all data is round-tripped through those cloud servers. This is stupid: if the cloud servers go away, the device is a brick. Yes, lots of companies do this, but part of the reason they do it is that they want to be able to control whether you can access a device you’ve bought by running the connection via their own servers, so they can charge you subscription money for it. If you’re not doing that, then the servers are a constant ongoing cost and you can’t ever shut them down. And it’s a poor model, and aggressively consumer-hostile, to require someone to continue paying you to use a thing they purchased. Not doing that. Communication should be local; the device is in my house, I’m in my house, why the hell should talking to it require going via a server on the other side of the world?

You can use a native app. Native apps can avoid the whole “this wifi network has no internet access so I will disconnect you from it for your own good” approach by calling various native APIs in the connectivity manager. A web app can’t do this. So you’re somewhat forced into using a native app even though you really shouldn’t have to.

Or you can use something other than Android; iOS, it seems, has a workaround although it’s a bit dodgy.

None of these are good answers. Currently I’m looking at building native apps, which I really don’t think I should have to do; this is exactly the sort of thing that the web should be good at, and is available on every platform and to everyone, and I can’t use the web for it because a bunch of decisions have been taken to prevent that. There are good reasons for those decisions, certainly; I want my phone to be helpful when I’m on some stupid hotel wifi with a signin. But it’s also breaking a perfectly legitimate use case and forcing me to use native apps rather than the web.

Unless I’m wrong? If I am… this is where you tell me how to do it. Something with a pleasant user experience, that non-technical people can easily do. If it doesn’t match that, I ain’t doin’ it, just to warn you. But if you know how this can be done to meet my list of criteria, I’m happy to listen.

I'm currently available for hire, to help you plan, architect, and build new systems, and for technical writing and articles. You can take a look at some projects I've worked on and some of my writing. If you'd like to talk about your upcoming project, do get in touch.

More in the discussion (powered by webmentions)

  • Gamer Geek responded at twitter.com Stuart Langridge: IoT devices and Android and disappointment kryogenix.org/days/2020/08/1…
  • Nicholas Shiell responded at twitter.com Sounds like a very frustrating problem :( I wounder how well Chrome handles being inside an old-school Intranet - as in a web app running inside a LAN…
  • Ben Thorp responded at twitter.com Can't believe I'm suggesting these, but what about either using WPS to connect your device to the router, or using WiFi Direct between your device and…
  • Stuart Langridge responded at twitter.com wifi direct is SPECTACULARLY undocumented. I cannot work out how to actually _do_ it. You also can't do it from a web app; you need a native app to ki…
  • Stuart Langridge responded at twitter.com I have no idea. I assume: badly :)
  • Stuart Langridge responded at twitter.com the problem with WPS is that it might not be your router; imagine you're in a cafe, for example, where you don't get to press the button on top of the…
  • Ben Thorp responded at twitter.com Ah. I was hoping that wifi direct could handle making a local network with the android device, and then serve the web app to the phone
  • Stuart Langridge responded at twitter.com you would think, wouldn't you? That's what I thought too. But either I'm too stupid to work that out, it's not actually possible, or it is possible an…
  • Ben Thorp responded at twitter.com Completely random third option - I wonder whether if you used your phone as a wifi hotspot that the device connected to, what you can serve over that…
  • Matthew Somerville responded at twitter.com Can your device fake the response to the request to connectivitycheck.gstatic.com?
  • Ben Thorp responded at twitter.com This guy seems to have got something working on an Arduino..... forum.arduino.cc/index.php?topi…
  • Stuart Langridge responded at twitter.com Yes, probably (that's how it's suggested that one does this on iOS). I'm concerned about that because then the phone will believe it's online and all…
  • Stuart Langridge responded at twitter.com that's not wifi direct. That's "using wifi, direct to the device"; half the problem with researching this topic is that people use the phrase "wifi di…
  • Ben Thorp responded at twitter.com
  • Andy Buckingham responded at twitter.com I'm almost certain you've thought about this, but, wouldn't the WiFi credentials be small enough to send over BLE? e.g. the browser handshakes, asks f…
  • Matthew Somerville responded at twitter.com The former, well, that’s the same as plenty of captive portals after they have connected :) The latter, given private networks etc, I don’t think it w…
  • Stuart Langridge responded at twitter.com yup. But you can't connect to a wifi network from a web page; you have to be a native app for that. Connecting to the device AP will disconnect becau…
  • Stuart Langridge responded at twitter.com interesting. Which Android version? Mine (10) disconnects, unless I put up a captive portal page, and then it disconnects if I close down the CPA brow…
  • Matthew Somerville responded at twitter.com Umm... 7.1.2. It may be called “Wi-Fi Assistant” or “Auto Network Switch” or “Smart Network Switch” or “Avoid Poor Connections”, loads of names it see…
  • ⊥ᵒᵚ Cᵸᵎᶺᵋᶫ∸ᵒᵘ responded at twitter.com I'm trying to build something which seems an ideal job for the web -- configuring a small local IoT device, where I shouldn't need a native
  • Stuart Langridge responded at twitter.com ah, it's been suggested by Neil McP that the "disconnect from no-internet wifi" thing is caused by (a) newer versions of android and (b) still having…
  • Ben Thorp responded at twitter.com
  • ⊥ᵒᵚ Cᵸᵎᶺᵋᶫ∸ᵒᵘ responded at twitter.com Android 8 certainly has a "use mobile data if Internet unavailable" toggle in WiFi settings. Would be OK as occasional set up thing to teach a new net…
  • MichaelP responded at twitter.com
  • Will Cooke responded at twitter.com
  • Andy Buckingham responded at twitter.com Ah, so is the issue that the web page that sets it up could disappear if the operator went bump?
  • Stuart Langridge responded at twitter.com that's a risk, but it'd be easy enough to fix for dedicated users. The problem is that having to teach the device about the wifi network in each place…
  • Dr. Roy Schestowitz (罗伊) responded at twitter.com #IoT devices and #Android and disappointment kryogenix.org/days/2020/08/1…
  • Šime Vidas responded at twitter.com I think I’ve found the best example yet of “technically good contrast but kinda hard to read.” (kryogenix.org/days/2020/08/1…)