Owntracks stops responding with a client certificate

[DConcord]

Hi, I believe I'm experiencing a similar/same issue to #525

Deployment details: MQTT over TLS (not http/websocket) on TCP 443 with username/password AND client certificates (Internal CA - Openssl). Static public IP assigned to the MQTT server in a DMZ via NAT, but the same public IP is used externally and internally

MQTT Server: Mosquitto 1.6.4
Relevant Mosquitto Configuration: "require_certificate true" and " use_identity_as_username false" in order use both username/password AND client certificate

Phone: XR on Owntracks 13.0.2 (Although I've seen the issue previously as well

Protocol: MQTT (not websocket) on TCP 443

Issue:
After about 8-12 hours, one of my two deployed s stops responding with a client certificate. Connection attempts continue, but in the Mosquitto logs I see:

OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate

In Owntracks, I open info (i) and see the following error:

idle The operation couldn't be completed. (OSStatus error -9829.) {
  "_kCFStreamErrorCodeKey" = "-9829";
  "_kCFStreamErrorDomainKey" = 3;
}

I just need to go further into settings and back out for everything to resume properly for another 8-12 hours

I do believe there's a correlation between switching between networks as the phone was working consistently from the same carrier IP address, then it switched to a wifi/terrestrial address, and after it tried to return to the same carrier IP address, the issue continued from then-on. All of these addresses are outside/external to my MQTT server:

(IPs and name sanitized for privacy)

!Cellular Carrier public IP - working properly
1571762440: New connection from 108.X.X.57 on port 443.
1571762440: New client connected from 108.X.X.57 as UserA (p2, c0, k60, u'UserA').
1571762455: Client UserA disconnected.
1571762695: New connection from 108.X.X.57 on port 443.
1571762695: New client connected from 108.X.X.57 as UserA (p2, c0, k60, u'UserA').
1571762710: Client UserA disconnected.
1571762995: New connection from 108.X.X.57 on port 443.
1571762995: New client connected from 108.X.X.57 as UserA (p2, c0, k60, u'UserA').
1571763010: Client UserA disconnected.
1571763051: New connection from 108.X.X.57 on port 443.
1571763052: New client connected from 108.X.X.57 as UserA (p2, c0, k60, u'UserA').
1571763066: Client UserA disconnected.

!Comcast/Terrestrial Wifi IP - last working connection
1571763362: New connection from 66.X.X.32 on port 443.
1571763362: New client connected from 66.X.X.32 as UserA (p2, c0, k60, u'UserA').
1571763377: Client UserA disconnected.

!Possibly unrelated phishing as this IP is from an unrecognized ISP in a different location
1571765490: New connection from 128.X.X.134 on port 443.
1571765490: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571765490: Socket error on client <unknown>, disconnecting.

!Back to the same Cellular Carrier public IP as before - no longer working properly from here, onward
1571770672: Outgoing messages are being dropped for client UserA.
1571771419: New connection from 108.X.X.57 on port 443.
1571771420: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571771420: Socket error on client <unknown>, disconnecting.
1571771502: New connection from 108.X.X.57 on port 443.
1571771503: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571771503: Socket error on client <unknown>, disconnecting.
1571771582: New connection from 108.X.X.57 on port 443.
1571771582: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571771582: Socket error on client <unknown>, disconnecting.
1571771802: New connection from 108.X.X.57 on port 443.
1571771802: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571771802: Socket error on client <unknown>, disconnecting.
1571771978: New connection from 108.X.X.57 on port 443.
1571771978: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571771978: Socket error on client <unknown>, disconnecting.

!Century Link ISP - I do believe this is the same client. Still external to the MQTT server
1571772103: New connection from 72.X.X.142 on port 443.
1571772103: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571772103: Socket error on client <unknown>, disconnecting.

Additional Info
I have two XRs deployed. One of them has this issue repeatedly, and other does not have the issue at all.

They both use Client certificates sequentially issued from the same CA/Issuing CA using the same cert template, and I actually used a variation of the same password on the PKCS certificate bundle that uses the same letters/symbols/length but in a different order. I've triple checked that both phones have imported and trusted the CA and Intermediate CA the same way into the Profile/certificate settings and I compared all standard and even advanced settings to make sure everything is identical. So its pretty puzzling why one works consistently and the other doesn't.

Everything works wonderfully when I disable "require_certificate" in Mosquitto, but I'd really love to be able to use this feature.

thanks!

[ckrey]

In the meantime we started to replace the underlying network layer with NSURLSession 7ecb822

Pls re-test when the new version becomes available

[DConcord]

Hey @ckrey Just revisited this and still having the same problem. Interestingly, I found this recent (2 mo old) thread reply by Apple support agent "eskimo" where he says that it's a known issue for NSURLSession but has been fixed WKWebView. Ignorance disclaimer alert!: is there any way WKWebView could be utilized rather than NSURLSession? Thanks!

developer question snippet:
Are client certificate authentication challenges in a background session still not supported as of iOS 13.3.1?

eskimo's answer:
Sadly, that’s still the case. This was fixed for a

WKWebView

a while back, but

NSURLSession

background sessions still suffer from this limitation (r. 17526855).

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

https://developer.apple.com/forums/thread/28713

[ckrey]

@DConcord thanks for following this up.

Via NSURLSession, NSURLSessionStreamTask / NSURLSessionWebSocketTask are used for MQTT / Websocket connections. There is no equivalent in WKWebView for that.

If you drop me your email, I will invite you to the beta testing channel to check if NSURLSession in iOS13 solves your problem.

[hanche]

Thanks for sending me the Testflight version (as you noted over on #642). Unfortunately, it never managed to connect. Status remained “connecting”, and on the server side, I got this in the log (real addresses replaced by IPv[46]:

20200824_201247: New connection from IPv4 on port 8883.
20200824_201247: New connection from IPv6 on port 8883.
20200824_201247: Socket error on client <unknown>, disconnecting.
20200824_201248: OpenSSL Error: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
20200824_201248: Socket error on client <unknown>, disconnecting.

[ckrey]

I guess you are using a self signed certificate on the server. In the iOS13 versions, you will have to install the server CA file in iOS to establish the trust. The OwnTracks option to suppress domain name validation is no longer used.

https://owntracks.org/booklet/features/tls/#ios

[ckrey]

I fixed another related problem. Version 13.3.4 should be available through Testflight soon

[hanche]

Well no, unless I understood you incorrectly. I have created my own CA certificate, which is installed on my phone and marked as Verified in nice green letters in the Settings app. And then I have created server and client certificates using that CA certificate: The former is used by the mosquitto server, and (one of) the latter in the OwnTracks app. Pretty sure I followed the instructions, and it does work in the most recent version on the App Store, just not in the Testflight version. I'll by sure to try again when a new Testflight version comes out, though.

Edited to add: I did not redo the setup steps with the Testflight version. I just assumed it would inherit everything from the App Store version, and looked briefly into Settings and checked that it all looked right.

[hanche]

No change for me with the latest testflight version. But I have no time to experiment with different settings right now. Maybe later. (Except I don't know what to change.)

[ckrey]

[hanche]

With that setting, it connects. But I feel uneasy about the security implications. Not that I think it likely that someone will try to trick owntracks on my phone to connect to their MQTT server instead of my own, but I'd prefer if they can't.
Anyhow, for now I'll keep it this way and see if it connects more reliably.

[ckrey]

You don't need to enable Allow Untrusted Certificates if your server's certificate does adhere to the new checking policy Apple introduced with iOS 13:

https://support.apple.com/en-us/HT210176

[jpmens]

FWIW, we've fixed our certificate generation utility to adhere to those rules: server certificates are issued with a validity of 825 days.